OpenSky Tech

Jonathan H. Wage: Ruler: A simple stateless production rules engine for PHP 5.3+

Mon, 10 Sep 2012 18:05:43 -0400

Jonathan H. Wage: Ruler: A simple stateless production rules engine for PHP 5.3+:

What is ruler?

Ruler is a simple stateless production rules engine for PHP 5.3+ written by Justin Hileman (@bobthecow). Justin was previously employed at OpenSky but these days you will find him hacking on a new startup named @presentate.

What is a rules engine?

From …

Jonathan H. Wage: Doctrine DBAL: PHP Database Abstraction Layer

Sun, 09 Sep 2012 12:25:50 -0400

Jonathan H. Wage: Doctrine DBAL: PHP Database Abstraction Layer:

jwage:

Most people think ORM when they hear the name Doctrine, but what most people don’t know, or forget, is that Doctrine is built on top of a very powerful Database Abstraction Layer that has been under development for over a decade. It’s history can be traced back to 1999 in a library named…

Jonathan H. Wage: Deploying OpenSky with Fabric

Sun, 09 Sep 2012 12:24:24 -0400

Jonathan H. Wage: Deploying OpenSky with Fabric:

jwage:

At OpenSky we use Fabric to deploy new versions of software to our servers. We deploy dozens of times a day to our testing environments, and do daily deploys to production.

Our production web nodes are split in to two groups, group1 and group2. It is setup that way so we can easily pull out…

Kris Wallsmith: Symfony2 ESI error: File name too long

Wed, 09 May 2012 12:44:55 -0400

Kris Wallsmith: Symfony2 ESI error: File name too long:

kriswallsmith:

We recently came across a strange, intermittent error at OpenSky:

(36)File name too long: Cannot map GET /_internal… HTTP/1.1 to file

We use Varnish to cache large sections of our pages that include follow/unfollow buttons. If the user is following a certain curator we show the unfollow…

Kris Wallsmith: Hello Spork! (aka "Forking PHP...")

Tue, 10 Apr 2012 13:31:39 -0400

Kris Wallsmith: Hello Spork! (aka "Forking PHP..."):

kriswallsmith:

A few months ago I was tasked with speeding up the upload of assets to the OpenSky CDN, which was taking a few minutes each deploy. I ended up dividing the upload into multiple processes using pcntl_fork() and bringing the total time of the upload down to a matter of seconds.

Since then I’ve…

Kris Wallsmith: I ♥ Event Delegation

Sat, 24 Mar 2012 11:21:11 -0400

Kris Wallsmith: I ♥ Event Delegation:

kriswallsmith:

jQuery event delegation is ~~one of the coolest things since~~ way cooler than sliced bread.

This new hotness allows you to listen for an event farther up the DOM than where that event is triggered. If you understand event bubbling, you should be able to grok this fairly quickly.

It starts getting…

"People tend to say things like ‘Git makes more sense when you understand […]’."

Mon, 07 Nov 2011 11:58:10 -0500

“People tend to say things like ‘Git makes more sense when you understand […]’.”

- Required reading of the day: Think like (a) Git.

Changing history, or How to Git pretty

Fri, 04 Nov 2011 22:07:08 -0400

Changing history, or How to Git pretty:

OpenSky’s engineering and product teams have an ongoing lunchtime presentation series called Lunch and Learn. A couple of weeks ago, I gave a talk entitled “Lunch and learn2git”. I just posted an article on my blog based on that presentation. Check it out!

OpenSky moving day!

Thu, 20 Oct 2011 12:08:00 -0400

OpenSky moving day!

How we chose Symfony2 as our web framework

Thu, 03 Feb 2011 11:00:00 -0500

As we set out to build the next great social ecommerce platform we were struck with a rare and great opportunity to start with a clean slate. We were able to select each piece of our architecture without any predefined requirements. This let us review the state of the Frameworks in PHP and select the best one for our needs.

First we developed a set of requirements

Brace yourself, it’s a long list.

ORM
DB Obj relationships ie: :has_many
Good db migration mgmt
Good master slave config for db
Intelligent db trans support
Scaffolding
CRUD friendly
Business logic kept in model not controller
Controllers have ability to get heavy
No heavy loading of controllers
No race conditions for controllers (ie routing)
Template engine
Objects accessible in view
Widget/component support in view layer
Good header mgmt
Good mgmt of js and css files
All js and css be compressed
Ability to define explict routes or priorities not based on file system
Smart SEO urls by default
XSS & CSRF Protection
Good Form builders
Form Validations
No XML => YML or JSON
enforcement of coding stds
Convention and configuration
Ability to design by contract
Caching: Support for memcache and sessions
Configs cached not parsed
Cloud sensitive
Good role base ACL support (no global, system base)
Support for webservices
Easily loadable support for external libs
Profiler built in or easy to integrate (timers, number of queries, memory used, etc)
Localization (currency, dates etc)
CLI model access
CLI for testing
Unit testing
Actions are unit testable
Good file handling
Good exception handling – ie log4j
Support for multiple environments (dev, test, staging, prod)

Comparing Frameworks

Then we looked at any framework that someone on the team would be willing to evaluate. We ended up with 7 Frameworks to compare.

We only evaluated PHP specific frameworks as our team is primarily a PHP team and that is where our expertise is.

Notice that Zend Framework, Cake, Code Ignitor or X Framework aren’t on the list. There were a good 10 or so popular frameworks that enough of our developers had used before and knew they didn’t meet our requirements, so we decided not to spend time evaluating them further.

Our Analysis

We performed this analysis May 2010. In the past 6 months both Symfony2 and Lithium which are under rapid development would score even better. It’s not my intention to keep this chart up to date as it’s a representation of historical analysis.

Further Analysis

At the end of our week of analysis we had a good overview of each Framework though there wasn’t a clear winner. What was apparent was that a “check” wasn’t equal to another check and some frameworks excelled in certain areas where other’s simply did enough to check the box. This made our analysis quite a bit harder.

Additionally there were some other factors that weighted heavily into our decision. Strength of community, Quality of development and Momentum are the three factors that played the biggest role in our decision making, but are not represented in this feature set analysis.

Ultimately we narrowed it down to 3 frameworks. Lithium, Symfony2 and Zoop2. All 3 were PHP 5.2+ and Next Generation Frameworks. Given the substantial changes accompanying PHP 5.3, we determined that either we went with that now, or had a painful migration in our future. We decided to bite the bullet now which left us to choose between Lithium and Symfony2.

Lithium vs Symfony2

This wasn’t an easy decision. Both are excellent frameworks developed by really smart people. Nate Abele, who started the Lithium project and was the lead developer of CakePHP, is’s a personal friend and an NYC local. He was the person who turned me, and consequently OpenSky, on to MongoDB. Nate is doing innovative things with Lithium and has a true passion for what his work.

Symfony2 is developed by Sensio Labs located primarily in France, on the other side of the world. At the time there were only 3 stateside Symfony team members. Symfony2 uses a bit more conservative approach to framework development, basing it largely on learnings from symfony 1 and established practices and patterns largely from the Java world.

Ultimately we went with Symfony2. It wasn’t a cut and dry decision and in spite of all our diligence still ended up as a bit of a gut call. Symfony boasted a larger developer base and community. This was probably the factor that ultimately pushed us in Symfony2′s direction.

Why Symfony2

Fast forward 9 months. OpenSky has embraced Symfony2 with open arms. We have become the 2nd largest contributors to Symfony2 behind Sensio Labs. We employ Kris Wallsmith and Jon Wage, two core Symfony2 team members. We’ve developed one of the first production applications using Symfony2 and the first ecommerce platform on MongoDB.

Inheritance in PHP, or why you want to use an Interface or an Abstract class…

Tue, 30 Nov 2010 10:38:00 -0500

Every class has an interface, every class can be typehinted as a method argument, typehint lets you specify the collaborator requirements. Not every typehint is the same… Example:

<?php

class PaymentGateway
{
    public function charge($amount, BankAccount $account)
    {
        // make a call to http web service here...
        $account->credit($amount)
    }
}

<?php

class BankAccount
{
    public function debit($amount)
    {
        //...
    }
    public function credit($amount)
    {
        //...
    }
}

The example above is fairly straightforward – we have a PaymentGateway class that requires a BankAccount collaborator for crediting amount to be charged in its charge function. Since we call credit() on the second argument – we typehint it to be of type BankAccount, which would prevent us from passing anything other than instances of that class. Let’s add some more requirements to our little example. Say we have checking and savings accounts, and we need to be able to charge both of them:

<?php

class CheckingAccount extends BankAccount
{
}

class SavingAccount extends BankAccount
{
}

Since both accounts are subclasses of BankAccount – the typehint still works fine. However, we have now opened the possibility for abusing the typehint… In case you didn’t know – classes in PHP don’t define a strict interface, and are open to change by subclasses:

<?php

class SomeAccount extends BankAccount
{
    public function credit($amount, $someOtherParameter)
    {
        //...
    }
}

NOTE: if you have strict errors enabled, you would get one…

In the above example we changed the credit method signature by adding a second required parameter. We can now pass instances of SomeAccount class everywhere the BankAccount is expected and get nice fatal errors. Only interfaces and abstract classes/methods lock method signatures.

<?php

interface BankAccountInterface
{
    function credit($amount);
    function debit($amount);
}

class BankAccount implements BankAccountInterface
{
    //...
}

For our particular example, we could use an abstract class:

<?php

abstract class BankAccount
{
    abstract public function debit($amount);
    abstract public function credit($amount);
}

Please note that in the example interface, I don’t specify method visibility. The reason is every method of the interface MUST be public and no explicit visibility means public in PHP.

<?php

class SomeAccount extends BankAccount
{
    public function credit($amount, $someOtherParameter)
    {
        //...
    }
}

A class like the one above would cause a fatal error to be thrown: “Declaration of SomeAccount::credit() must be compatible with that of BankAccountInterface::credit()” or “Declaration of SomeAccount::credit() must be compatible with that of BankAccount::credit()”, depending on which approach you chose to use. If someone wanted to implement a BankAccount to be used with our PaymentGateway, they would have to keep the method signatures exactly the same as we defined them.

Happy Coding!

ZendCon 2010 in Santa Clara, California

Mon, 08 Nov 2010 16:12:00 -0500

Hi everyone, in case you didn’t know – there was a ZendCon last Monday November 1st through Friday November 5th, 2010 in Santa Clara, California and most importantly – I got to attend and OpenSky sponsored my trip.

ZendCon is an annual conference organized by Zend the PHP company and held in California. Its known to be one of the biggest conferences in PHP world with attendees from all over the world varying from developers to top management, all using and interested in PHP. Zend makes PHP announcements – this year among other things, they announced that memory footprint of PHP had been reduced by 35% in the upcoming version – which are then followed up by different session from speakers from all over the world. These sessions vary in scope and context from detailed practical “how to”s of a framework or tool, to high level management advices and experiences.

As I already mentioned, this year I was thrilled to be among ZendCon attendees!

I arrived in Santa Clara on the evening of November 1st. This was my very first visit to west coast and I loved it. The first thing I saw when arrived at Santa Clara (after a 6 hour flight and a 2 hour ride on three trains – one of which had two levels) was a big Yahoo sign on one of the buildings which immediately felt like technology heaven.

During the next three days, I met interesting people from all over the world, including people I knew from twitter, but never met in person. I would like to especially thank Ryan Weaver and Pablo Godel, who are seasoned conference attendees, for making a great company and helping me get accustomed to the environment and people at the conference. I also met and spent a decent amount of time with people as: Jonathan Wage, Fabien Potencier, Nils Aderman, Dustin Whitle, Michelangelo van Dam, Jeremy Kendall and many many others.

I learned about Zend Uncon from Michelangelo during continental breakfast of the very first day of the main conference. Turns out that anyone could give a talk, even if it wasn’t included into the main session program. This is called the Uncon and there you can give a talk, do an interactive tutorial or round table on any topic you choose, all you need is to put your name in an empty slot in the Uncon schedule. Uncon sounded like a great idea and a good public speaking practice, so I registered… The session was amazing! Twitter was blown by quotations from my talk - tweeted, retweeted and re-retweeted. Attendees took pictures of me speaking and posted it on twitter too. Overall I was blown away by the success of my little experiment. As a result, my session was called the best Uncon session and I won a free lifetime license for Zend Studio 8 (they even wanted to call me up on stage during the final keynote, but I had a plane to catch that day).

Jonathan Wage, who joined OpenSky on the November 1st, 2010, suffered from poor organization on the conference part. He was unfortunate to get scheduled to arrive after his first session and leave before the second one. As a result, he gave only one talk of the two planned as part of the Uncon. His talk was, however, really amazing and well received by attendees and they wouldn’t let him go for a good 30 minutes after the talk was over, asking questions and expressing admiration.

Thank you ZendCon and see you again next year! Below are some pictures I took at the conference:

by avalanche123

Unit Tests, Mocking, and PHPUnit 3.5′s new Mock Builder

Fri, 29 Oct 2010 16:00:00 -0400

Unit testing is all about testing one piece of functionality in isolation. This is very easy when the subject of your test doesn’t involve any dependencies. Take the following method:

Unit Tests, Mocking, and PHPUnit 3.5′s new Mock Builder

Fri, 29 Oct 2010 16:00:00 -0400

Unit testing is all about testing one piece of functionality in isolation. This is very easy when the subject of your test doesn’t involve any dependencies. Take the following method:

	<?php

	class Tag
	{
	    public function normalizeName($name)
	    {
	        $normal = trim($name);
	        $normal = strtolower($normal);

	        return $normal;
	    }
	}

Testing this method requires just an input and an expected output. This is a good case for using a PHPUnit dataProvider:

	<?php

	class TagTest extends PHPUnit_Framework_TestCase
	{
	    /**
	* @covers Tag::normalizeName
	* @dataProvider provideNormalizeName
	*/
	    public function testNormalizeName($input, $expected)
	    {
	        $tag = new Tag();
	        $this->assertEquals($expected, $tag->normalizeName($input));
	    }

	    public function provideNormalizeName()
	    {
	        return array(
	            array('foo', 'foo'),
	            array('FOO', 'foo'),
	            array(' foo ', 'foo'),
	            array(' FOO ', 'foo'),
	        );
	    }
	}

Unfortunately, it’s not always this easy. Let’s say we want to centralize this normalization logic in some sort of reusable Normalizer class, which we inject into the Tag class:

	<?php

	class Tag
	{
	    protected $normalizer;

	    public function __construct(Normalizer $normalizer)
	    {
	        $this->normalizer = $normalizer;
	    }

	    public function normalizeName($name)
	    {
	        return $this->normalizer->normalize($name);
	    }
	}

The test above still passes, but we are now testing two things: the normalizeTag method and the Normalizer class; we are no longer testing our method in isolation.

Mock objects to the rescue!

To restore this isolation we create a “mock” version of the Normalizer class and tell that mock exactly what we expect will be done to it and how we want it to respond.

In PHPUnit 3.4, creating this mock object would look something like this:

	<?php

	class TagTest extends PHPUnit_Framework_TestCase
	{
	    /**
	* @covers Tag::normalizeName
	*/
	    public function testNormalizeName()
	    {
	        $input = 'foo';
	        $expected = 'foo';

	        // the final "false" tells PHPUnit not to call the Normalizer
	        // constructor because it requires another dependency, which we don't
	        // care about right now
	        $normalizer = $this->getMock('Normalizer', array(), array(), '', false);
	        $normalizer
	            ->expects($this->once())
	            ->method('normalize')
	            ->with($input)
	            ->will($this->returnValue($expected));

	        $tag = new Tag($normalizer);
	        $this->assertEquals($expected, $tag->normalizeName($input));
	    }
	}

Most of this is quite readable, thanks to Sebastian’s nice mocking API. However, the initial getMock method is a bit gnarly. I’ve added a comment so you understand what’s going on, which I’d rather not have to do.

In PHPUnit 3.5 we can use the new mock builder object, a contribution of Giorgio Sironi, which makes our test code that much more readable:

	<?php

	class TagTest extends PHPUnit_Framework_TestCase
	{
	    /**
	* @covers Tag::normalizeName
	*/
	    public function testNormalizeName()
	    {
	        $input = 'foo';
	        $expected = 'foo';

	        $normalizer = $this->getMockBuilder('Normalizer')
	            ->disableOriginalConstructor()
	            ->getMock();
	        $normalizer
	            ->expects($this->once())
	            ->method('normalize')
	            ->with($input)
	            ->will($this->returnValue($expected));

	        $tag = new Tag($normalizer);
	        $this->assertEquals($expected, $tag->normalizeName($input));
	    }
	}

So there you go: unit tests, mocking, and the new mock builder.

by Kris Wallsmith

Symfony NYC Meetup: Unit Testing

Tue, 26 Oct 2010 09:00:00 -0400

People that RSVPed for the last Symfony NYC Meetup know, that it got canceled. We are very sorry, but the meetup speaker got sick and couldn’t make it. He did promise to give his talk on ESI caching as soon as he feels better.

Meanwhile, the meetup was moved to this coming Thursday, October 28th, 2010. The new topic is “Unit Testing. A guide to writing clean, testable code, that will be easy to maintain and extend”. This will be the repeat of the talk I gave to the OpenSky tech team a month ago. I will cover topics like unit-testing, dependency injection, mock objects and ways to refactor your code.

If you haven’t yet, please RSVP for the meetup and more importantly, come to hear me speak on the topic.

Edit: I uploaded the slides from that talk to SlideShare and embedded them below

Clean code

View more presentations from Bulat Shakirzyanov

Edit 2: We video-taped the full talk, which you can watch by clicking on the link below

by avalanche123

Clean code talk on Friday

Wed, 29 Sep 2010 10:00:00 -0400

At OpenSky, we concentrate on programmers’ skills and ways to improve them a lot. As part of that, we have Fridays dedicated to skill improvement – we, programmers, are allowed to write blog posts, prepare presentations¸ educate others on what we learned, refactor/improve our code or simply go through and add tests where they are missing.

Last Friday was very special. I finally took the time (the whole weekend before that) and put together some slides to give a long requested talk on unit-testing and the art of programming.

I wanted to cover as many aspects of testing and OOP in general as I possibly could, while preserving the structure of test-oriented talk, without concentrating too much on any single area.

I had no idea how to accomplish that when I started. However, the introduction section had given me a good understanding of how to relate all the different aspects, so it didn’t feel like I was jumping from topic to topic, not covering each in enough detail. I structured the talk around the properties of unit test – the properties we want to achieve in order to practice unit-testing. As a result of that, I was able to cover Dependency Injection (with the Law Of Demeter as a natural consequence), Mock Objects (leading me to Iterative Interface Discovery and Test Driven Development) and some general OOP techniques, that let programmers achieve much desired code clarity (Inheritance instead of Conditionals and Composition over Inheritance).

Ideas promoted in the talk were inspired by a number of different sources, and I feel I should list the references, since they were not included in the slides:

All definitions were taken from Wikipedia.org
The Clean Code Talks — Unit Testing by Misko Hevery (Google Inc.)
Mock Roles, Not Objects by Steve Freeman, Tim Mackinnon, Nat Pryce, Joe Walnes (ThoughtWorks UK)
The Clean Code Talks — Inheritance, Polymorphism, & Testing by Misko Hevery (Google Inc.)

The talk turned out to be pretty good and I received a lot of positive feedback on it from outside after posting it on slideshare. Even though I reserved 90 minutes to cover everything and answer all the questions, I was not able to do so, since we started later than planned. Therefore, I’m sharing the talk here and if anyone has questions or comments after watching the slides, feel free to post them here.

by avalanche123

Symfony NYC Meetup

Tue, 28 Sep 2010 15:33:00 -0400

Some of you might, others might not know that our company, OpenSky, became the official sponsor of Symfony NYC Meetups, which means, that we’ll be seeing you the last Thursday of every month, here, at OpenSky HQ in NYC for the foreseeable future.

That was enough of the introduction, now to the point. Last Thursday, September 23rd, 2010, OpenSky held its first Symfony NYC Meetup. And, as anything OpenSky does, the Meetup turned out amazing. Our very own Kris Wallsmith gave an awesome talk, highlighting the differences between symfony 1.x and Symfony2. This was particularly useful, since my co-workers are the only people I know in NYC using Symfony2, and the majority of Meetup attendees are still in symfony 1.x world and hesitate to switch.

During the second, less structured part of the Meetup, we were enjoying pizza, that OpenSky kindly provided, discussing Symfony2 feature roadmap, current limitations of certain components and proceeded to other interesting, but hardly Symfony2 related topics like NodeJS, Android OS, Document-Oriented Databases (CouchDB and MongoDB) and PHP 5.3 Object Document Mappers (Doctrine MongoDB ODM and Doctrine CouchDB ODM).

In general it was a very interesting event and I feel sorry for those who couldn’t attend.

Since this is definitely not the last Meetup OpenSky is hosting, make sure to come in next time, to meet fellow symfony developers, share ideas and get to know OpenSky.

As a bonus, here is Kris’s video, where he talks about Symfony2.

by avalanche123

Symfony2 PR 3 released

Mon, 27 Sep 2010 10:00:00 -0400

The core symfony 2 team has just released Symfony 2 Preview Release 3. Of the 50+ different contributors, OpenSky provided 5 of them. Our developers contributed meaningful code commits which the core team approved and integrated with the release. We are excited to be a part of this thriving community and to be able to both use and contribute to such an excellent framework.

Here is a list of our team members and their commit counts towards this release:

39 Kris Wallsmith
16 Jeremy Mikola
6 Bulat Shakirzyanov (avalanche123)
2 Justin Hileman
1 Matt Fitzgerald (tiranog06)

http://www.symfony-project.org/blog/2010/09/13/symfony2-pr3-released-the-need-for-help

by Steve Francia

I Sell Tons of Products with the Term “beauty” in them. Why don’t I show up in the Search tab for Sellers?

Tue, 21 Sep 2010 10:00:00 -0400

This article was written by Matthew Fitzgerald, Software Engineer, and Mike Zupan, Systems Engineer at OpenSky

The fix will go live next week (Sept 29)

One of the most important ways Shoppers find Sellers is through Search. Recently, we’ve found a hole in our search index. The problem we had was, Sellers would select Products with keywords like “beauty” in the Product Name but that Seller would not show up in the Search results for “beauty”.

The reason for this was our Seller search index only had direct information on the Seller, ie: Seller Username, Business Name, Bio, etc… The Seller index had no knowledge of Products the Seller was promoting. So any keywords the Product might have, wouldn’t return the seller in a search.

The Solution

We needed to modify our Seller index to have knowledge of both Products and Sellers and return the results of distinct sellers. The problem, our current Search technology (SOLR) didn’t support that behavior of Grouping Product Records by Seller and collapsing them into distinct Seller results.

The Tech Part

After some hefty Googling, I found that the feature to Group Search results does exist in a future version of the technology yet to be released. This feature became so popular that the developers of our Search technology released a “patch” to include this feature in the current version of the software. This is how I applied the patch.

Step 1

If you are like me, you hate Subversion and you love GIT. I run Ubuntu and using apt-get I installed a git subversion bridge.

sudo apt-get install git-svn

Step 2

Get some reading material, then start the checkout the latest stable tag of Solr

git-svn clone http://svn.apache.org/repos/asf/lucene/solr/tags/release-1.4.1

Step 3

Download and apply the patch

cd release-1.4.1
wget https://issues.apache.org/jira/secure/attachment/12448216/SOLR-236-1_4_1.patch
patch -p 0 -i SOLR-236-1_4_1.patch

Step 4

Recompile it, Baby!

ant compile

Now you want to build the example project so we can test it

ant example

Now it is time to test. So go into the example project and the example data directory

cd example/exampledocs

Now you can import all the xml data

java -jar post.jar *.xml

Now start the server

cd ..
java -jar start.jar

Now you should be able to hit the following url and get back grouped data

http://localhost:8983/solr/select?

  wt=json&indent=true&fl=id,name
  &q=solr+memory
  &group=true&group.field=manu_exact
  &collapse.field=manu_exact

Are you into PHP? This is how to implement it with the native PECL Driver.

$query = new SolrQuery('beauty');
$query->addParam('group','true');
$query->addParam('group.field','sellerId');
$query->addParam('collapse.field','sellerId');
$query->addParam('rows', 1000);

by Matthew Fitzgerald

Creating a unified search experience

Sat, 18 Sep 2010 10:00:00 -0400

If you’ve ever used the Internet (how else are you reading this?), then you have used a search engine. Search powers everything from shopping to dating to research, and is so common that people use it without thinking about it… it is just natural.

Of course, when something is natural, you expect a comfortable experience and a good fit. You probably don’t wake up in the morning and put on someone else’s clothes; instead, you probably have a wardrobe of comfortable apparel that you piece together a look that feels good and feels right.

We don’t feel this is how our search experience at OpenSky has been. If you wanted to find products, you could get to them pretty well. But if you were a seller trying to find a great supplier to connect with, the experience was… lacking? discombobulated? Not cohesive, at any rate.

So we decided it was time to roll our own search experience rather than let our community search continue to be powered by [insert ginormous search engine]. Here is how we did it.

Search Technology

We use Apache Solr to power our custom searches. Solr is a search platform that utilizes the popular Lucene search library as its core, is blazing fast, and is open-source… hey, we like open source projects around here.

We decided to create a new Solr index for indexing and searching community data, rather than create a unified index for searching all data, the primary reasons being variance in relevant data fields and the ability to apply the unique schema to do more complex, targeted, and nifty search applications that we haven’t thought up yet.

There are certainly some benefits that can be gained from having a unified index, and we are currently cooking up some things which will make use of both unified and specific indexes, so we can make our search experience as useful and intuitive as possible. Expect more great things coming up as search continues to evolve.

Schema Considerations

Careful thought and consideration went into how we designed the schema. We are using the DisMax Query Parser for our general search, and one of the things we learned that hard way is that inconsistent field configuration options, such as stopwords or other tokenizer index / query filters, can seriously affect query results. Also, it pays off to tweak some default configuration options to your needs.

As a concrete example, the DisMax Query Parser defaults to requiring all search terms to be present in a field to generate a positive hit. Assume you have a document with title = “Essence of murtlap”, description = “Essence of murtlap is phenomenal!” Assume title is a “string” field with no tokenizer filters defined, while description is a “text” field with a stopwords configuration that prevents “of” from being tokenized, as it has high frequency and low importance.

In this case, a search for “Essence of murtlap” will not return any results. Why? Since “of” was not tokenized in the description field, 100% of the terms do not match, and so the entire document is a negative hit. However, searching for “Essence murtlap” would match. This may seem odd — after all, 100% of the terms DID match in the title field. For a mor thorough explanation, see one of several available resources.

I also decided to throw a tokenizer filter to remove leading and trailing punctuation. The inspiration for this was trying to match a word at the end of a sentence. Using the same example above, one would expect to be able to have a positive hit by searching for “phenomenal” — but one would be mistaken. Without some TLC the actual token stored is “phenomenal!” so, unless you include that punctuation in your search, you are going to be disappointed.

Fortunately, I found a very useful thread which presented a simple way to solve the problem using regular expression string filtering while indexing the document:

<filter
  pattern="^(\p{Punct}*)(.*?)(\p{Punct}*)$"
  replacement="$2"
/>

Conclusion

If you decide to use Solr to facilitate your next searching project, don’t be afraid to get your hands dirty and play with the schema configuration. Solr has a lot of good documentation, is extremely powerful, and isn’t nearly as scary as it seems.

In the meantime, enjoy the new unified search experience and keep looking for more great things to come.

by Ben Loveridge