Blog

I was fortunate to be able to speak last week (along with Joe Streeky, my Search Infrastructure Development Manager) at the very first Atlanta Solr Meetup held at Atlanta Tech Village. The talk covered how we scale Solr at CareerBuilder to power our recommendation engine, semantic search platform, and big data analytics products. Thanks to everyone who came out for a great event and to LucidWorks who sponsored us with the meeting place, pizza, and drinks.

Video:

Slides:
https://www.slideshare.net/treygrainger/scaling-recommendations-semantic-search-data-analytics-with-solr

Talk Summary: CareerBuilder uses Solr to power their recommendation engine, semantic search, and data analytics products. They maintain an infrastructure of hundreds of Solr servers, holding over a billion documents and serving over a million queries an hour across thousands of unique search indexes. Come learn how CareerBuilder has integrated Solr into their technology platform (with assistance from Hadoop, Cassandra, and RabbitMQ) and walk through api and code examples to see how you can use Solr to implement your own real-time recommendation engine, semantic search, and data analytics solutions.

Paper Abstract:
Common difficulties like the cold-start problem and a lack of sufficient information about users due to their limited interactions have been major challenges for most recommender systems (RS). To overcome these challenges and many similar ones that result in low accuracy (precision and recall) recommendations, we propose a novel system that extracts semantically-related search keywords based on the aggregate behavioral data of many users. These semantically-related search keywords can be used to substantially increase the amount of knowledge about a specific user’s interests based upon even a few searches and thus improve the accuracy of the RS. The proposed system is capable of mining aggregate user search logs to discover semantic relationships between key phrases in a manner that is language agnostic, human understandable, and virtually noise-free. These semantically related keywords are obtained by looking at the links between queries of similar users which, we believe, represent a largely untapped source for discovering latent semantic relationships between search terms.

Published in the Proceedings of the 8th ACM Conference on Recommender Systems (ACM RecSys 2014)

Paper:
https://arxiv.org/pdf/1409.2530v1.pdf

Timothy Potter and I were recently interviewed about the launch of our new book, Solr in Action, which was published last month. If you want to learn more about the book or just hear about our two-year journey to bring what critics are calling the “definitive guide” to Solr to market, please check out the podcast below:

Description:
This week, the SolrCluster team is joined by Trey Grainger, Director of Engineering for Search at CareerBuilder, and Timothy Potter, Lucene/Solr Committer and senior engineer at LucidWorks, to discuss their recently released co-authored book, Solr in Action. Solr in Action is a comprehensive guide to implementing scalable search with Lucene/Solr, based on the real-world applications that Tim and Trey have worked on throughout the course of their careers in Solr. Tim and Trey share with us the challenges they faced, accomplishments they achieved, and what they learned in the process of co-authoring their first book.

After nearly two years of writing, editing, and coding up examples, I’m excited to announce that Solr in Action has finally been published! We released our first “early access” version back in October of 2012 and have since been working tirelessly to round out this comprehensive (664 pages!) guide covering versions through Solr 4.7.

Solr in Action is an essential resource for implementing fast and scalable search using Apache Solr. It uses well-documented examples ranging from basic keyword searching to scaling a system for billions of documents and queries. With this book, you’ll gain a deep understanding of how to implement core Solr capabilities such as faceted navigation through search results, matched snippet highlighting, field collapsing and search results grouping, spell-checking, query autocomplete, querying by functions, and more. You’ll also see how to take Solr to the next level, with deep coverage of large-scale production use cases, sophisticated multilingual search, complex query operations, and advanced relevancy tuning strategies.

Solr in Action is intentionally designed to be a learning guide as opposed to a reference manual. It builds from an initial introduction to Solr all the way to advanced topics such as implementing a predictive search experience, writing your own Solr plugins for function queries and multilingual text analysis, using Solr for big data analytics, and even building your own Solr-based recommendation engine.

The book uses fun real-world examples, including analyzing the text of tweets, searching and faceting on restaurants, grouping similar items in an ecommerce application, highlighting interesting keywords in UFO sighting reports, and even building a personalized job search experience. Executable code for all examples is included with the book, and several chapters are available for free at the publisher’s website.

I just got back from a fantastic trip to Dublin, Ireland for last week’s Lucene/Solr Revolution EU. I was privileged this year to to present a deep dive (75 minute) session on “Enhancing Relevancy through Personalization & Semantic Search.” I appreciate all the great questions and feedback from everyone who attended.

Video:

Slides:
https://www.slideshare.net/treygrainger/enhancing-relevancy-through-personalization-semantic-search-28741313

Talk Summary: Matching keywords is just step one in the effort to maximize the relevancy of your search platform. In this talk, you’ll learn how to implement advanced relevancy techniques which enable your search platform to “learn” from your content and users’ behavior.

Topics will include automatic synonym discovery, latent semantic indexing, payload scoring, document-to-document searching, foreground vs. background corpus analysis for interesting term extraction, collaborative filtering, and mining user behavior to drive geographically and conceptually personalized search results.

You’ll learn how CareerBuilder has enhanced Solr (also utilizing Hadoop) to dynamically discover relationships between data and behavior, and how you can implement similar techniques to greatly enhance the relevancy of your search platform.

I just made it back from the beautiful, sunny city of San Diego where LucidWorks hosted another fantastic Lucene/Solr Revolution conference this week. I was invited back this year to present on “Building a Real-time, Big Data Analytics Platform with Solr.” Thank you to everyone who came and packed out the room, especially those who provided great feedback afterward and asked all of the terrific questions!

Video:

Slides: https://www.slideshare.net/treygrainger/building-a-real-time-big-data-analytics-platform-with-solr

Talk Summary: Having “big data” is great, but turning that data into actionable intelligence is where the real value lies. This talk will demonstrate how you can use Solr to build a highly scalable data analytics engine to enable customers to engage in lightning fast, real-time knowledge discovery.

At CareerBuilder, we utilize these techniques to report the supply and demand of the labor force, compensation trends, customer performance metrics, and many live internal platform analytics. You will walk away from this talk with an advanced understanding of faceting, including pivot-faceting, geo/radius faceting, time-series faceting, function faceting, and multi-select faceting. You’ll also get a sneak peak at some new faceting capabilities just wrapping up development including distributed pivot facets and percentile/stats faceting, which will be open-sourced.

The presentation will be a technical tutorial, along with real-world use-cases and data visualizations. After this talk, you’ll never see Solr as just a text search engine again.

I’m excited to announce early access availability of Solr in Action, a book on Apache Solr 4 which I am co-authoring with Timothy Potter. The MEAP (Manning Early Access Program) released today, which means that you can purchase the book early and receive new chapters as they are being written so that you don’t have to wait for the final release before having access. Three chapters are currently available (“Introduction to Solr”, “Key Solr Concepts”, and “Indexing”), and we expect a new chapter to be released every few weeks.

Please consider heading over to solrinaction.com and picking up a copy today!

Book Summary:
Whether you’re handling big data, building cloud-based services, or developing multi-tenant web applications, it’s vital to have a fast, reliable search solution. Apache Solr is a scalable and ready-to-deploy open-source full-text search engine powered by Lucene. It offers key features like multi-lingual keyword searching, faceted search, intelligent matching, and relevancy weighting right out of the box. Solr 4 provides new features to enable large-scale distributed search solutions that can be deployed as an elastically scaling cloud-based service and can provide additional intelligence to other big data technologies like Hadoop and Mahout.

Solr in Action is a comprehensive guide to implementing scalable search using Apache Solr 4. This clearly-written book walks you through well-documented examples ranging from basic keyword searching to scaling a system for billions of documents and queries. You’ll gain a deep understanding of how to implement core Solr capabilities such as faceted navigation through search results, matched snippet highlighting, field collapsing and search results grouping, spell checking, query auto-complete, querying by functions, and geo-spatial searching.

Along the way, you’ll discover more advanced topics, such as scaling Solr for large production environments, best practices and strategies for handling multi-lingual content, building a Solr-powered recommendation engine, performing complex data analytics, and integrating Solr with other big data technologies for machine learning and knowledge discovery.

You will also learn how Solr’s relevancy algorithm works, best practices and tricks for tuning and measuring your search relevancy, and even how to write and integrate Solr plugins and patches to introduce your own great new search features.

After a wonderful time last year, I was able to present yet again this year’s Lucene Revolution conference in Boston.  Lucene Revolution is a yearly conference put on by Lucid Imagination, a company focused upon supporting and commercializing the open source Apache Lucene and Solr search technologies (and integrating them with related technologies). My talk was entitled “Building a Real-time, Solr-powered Recommendation engine.”

Video:

Slides: https://www.slideshare.net/treygrainger/building-a-real-time-solrpowered-recommendation-engine

Talk Summary: Searching text is what Solr is known for, but did you know that many companies receive an equal or greater business impact through implementing a recommendation engine in addition to their text search capabilities? With a few tweaks, Solr (or Lucene) can also serve as a full featured recommendation engine. Machine learning libraries like Apache Mahout provide excellent behavior-based, off-line recommendation algorithms, but what if you want more control? This talk will demonstrate how to effectively utilize Solr to perform collaborative filtering (users who liked this also liked…), categorical classification and subsequent hierarchical-based recommendations, as well as related-concept extraction and concept based recommendations. Sound difficult? It’s not. Come learn step-by-step how to create a powerful real-time recommendation engine using Apache Solr and see some real-world examples of some of these strategies in action.

I recently gave a presentation at Lucene Revolution 2011 out in San Francisco.  The title of my topic was “Extending Solr: Building a Cloud-like Knowledge Discovery Platform.”

Video:

Slides: https://www.slideshare.net/treygrainger/extending-solr-building-cloudlike-knowledge-discovery-platform

Talk Summary: For CareerBuilder, a 1% deviance in search relevancy can mean millions of missed job opportunities for our users. When CareerBuilder moved to Solr from an expensive, proprietary search vendor, our top priorities were maintaining the quality of our search results and drastically improving our agility. This talk will describe how we addressed both needs. For search quality, we’ll cover some of our internal studies and resulting methods for dealing with multi-lingual content across dozens of languages, as well as customizing and experimenting with relevancy calculations. For platform agility, we’ll discuss CareerBuilder’s cloud-like search API framework which seamlessly handles millions of searches an hour, processes hundreds of millions of documents, and is powered by hundreds of globally-distributed servers. Come hear the results of our studies and some best practices for quality and performance. Learn how our framework has lead to staggering improvements in both maintainability and technology innovation, allowing us to learn from our content, not just find it.

Check out my interview, published today, with Mitch Pronschinske from DZone:  The Solr Conversion at CareerBuilder.com: Lower Costs, Greater Agility

Questions from the Interview:

• Jobs are one of the most important things we search for on the web.  What are some of the major challenges for search technology on a jobs site?
• What are some of CareerBuilder’s unique challenges in search?
• You led the conversion of CareerBuilder’s search platform from FAST ESP to Apache Solr.  Why did you think this was necessary and how did you convince upper management to make the change?
• What benefits have resulted from the switch to Solr?
• What were some of your search experiences related to genetic algorithms?
• Can you tell us about the cloud-like search API you created for CareerBuilder?
• Tell us about your side project, Celiaccess.com.