Caching Array from DB (MongoDB) in Node.js / Express.js

Question!

I wanted to add user search auto-complete (like Facebook's) to my Rails app on Heroku, and I chose to write it in Node.js because of the concurrency requirements. The search first pulls a user's friend list (of id's, which include all twitter friends, not just their friends on our site) from Mongo, then searches for users in that list, then does another search for any other users that match the query that weren't in the results returned by the friends search.

This was fairly fast at first (~150 ms), but for users with more friends (above, say, 100 total), loading their friends array ended up being a huge bottleneck, linearly slowing down the search to a maximum of around 1500 ms for user's with 1,000 friends (the maximum number supported for autocomplete friends search).

The problem is, I'm completely new to Node.js and Express (its Sinatra-like web framework), and I have no idea how to cache the friends array so I only need to load it once (ideally into memory). In Rails on Heroku I'd simply load the array into Memcache, but I'm not even sure how to configure Memcache in Node/Express, let alone if that's supported on Heroku.

Any ideas?

(Also note, I'm multi-key indexing for all these queries, including the friends ids)



Answers

I would suggest, if you aren't going to preload the names all into the client side then you would be better doing the search after the first character is input. This will reduce the number of names you need to search for to a fraction and then submit that request to the DB. You can then return those results in alphabetical order, once as more characters are typed you can filter without sorting. Every request should then meet your 150ms target as long as a user doesn't have thousands of friends called "David Smith".



This video can help you solving your question :)
By: admin