The online seller of shoes and apparel said a self-learning algorithm has shown promise in solving one of its most perplexing search-engine issues: irrelevant results.