MongoDB Performance: Group vs. Find vs. MapReduce

Lately, I've been playing a lot with MongoDB's group command. It's a powerful command that allows you to write some really interesting queries quickly. Here's an example of one that calculates "Tags" on a blog site:


In my tests it also proved to be about 4x faster than a similar MapReduce, however, it comes with a severe cost: It blocks all reading from the collection. This is a huge problem and basically makes it worthless for doing serious queries on a database with say hundreds of thousands of users like I have in my day job. From what I can tell finds, distincts, and mapReduces don't block and some combination of those provide non-blocking alternatives.

Here's a simple map reduce example:


One more note, I recently read MongoDB: The Definitive Guide, which had a lot of examples and clarification that are not readily available in the online documentation. I highly suggest checking it out!

Mobile vs. Native

I just wanted to comment on Ray C. Morgan's JSConf 2011 talk about web vs. native when it comes to mobile apps.

He started off his talk with a 15 minutes Objective-C lesson, which probably threw pretty much everyone off, but it was perfect for me. I recently had a client ask me to convert a mobile web app I had developed for them to a native applications, so I was knee deep in Objective-C. The talk really hit on a lot of things I had been thinking about and I'd like to re-iterate some of his points and then take issue with a few others.

Building Native Apps is Not that Hard

This maybe more than anything stuck out to me as I was learning Objective-C at the moment, because as a web programmer trying to learn how to do native iPhone apps, I could pretty much see where he was coming from. It's not impossible to learn Objective-C, and your apps are going to have some instant speed benefits as well as feel like native apps. People are used to these and if you're building apps you shouldn't assume that you can't do native even if you don't know how to do it yet. My experience with Android tells me pretty much the same things. It's definitely possible, even for us web-devs.

Build web pages in semantic HTML

While he kept saying "remember the good ol' days of semantic html?", and I think most people were thinking, no actually, we're still getting there. I think his point was pretty valid. If you're building a website, mobile or not, start with HTML, not a super fancy fake mobile framework. People expect web pages to act like webpages, so think about your mobile website the same way. It's not a mobile application, it's a web page. This gives you the best likelihood of your site being able to work on multiple mobile platforms and for the most part lets you take advantage of what you already know. Even media queries can be used really effectively in a lot of situations. Again, think about what people are doing with your site and what you're trying to accomplish.

Missing from his talk: Use fancy frameworks for prototyping

If you're a web developer and you want to do a proof of concept of a mobile application before you invest heavily in learning native language, use something like Sencha Touch to build it. It will look pretty native and give you a good idea if it's worth building. This is basically what happened to me, when I initially decided to build a Sencha Touch app for a client, who had an iPhone 3GS. The app is great and usable, but ultimately we decided to make it native. I could never have gotten so far along without starting the JavaScript framework, so I highly recommend using them as a prototyping tool.

Overall, I loved his talk and appreciated some of the points that were brought up. Yay for JSConf. Hopefully, I'll get around to posting more re-caps like this soon!

Mongo Capped Collections

We have a huge capped collection over here at i.tv that we're using for logging purposes. It's pretty great, except that it's so big that it's too slow to query, so
here's what you can do to get useful data out of the system:

db.requests.find().skip(10000).limit(300).toArray()

Capped collections don't usually have an index, so they're not good for searching when they get large, (you set how big you want them), but they are in chronological order by default, so our little query will find the records between 10,000 to 10,300 which will correspond with an actual block of time.

It's not a bad way to get information out of the system, but beware, because the logs are always growing, you'll need to move your skip back relatively frequently.

no.de, github, one repo

Are you like me and use github for your repo, but also use Joyent's awesome no.de service to host your website. That's exactly what I'm doing with this CollabPaintJS project I'm working on.

It was silly to have two separate local directories when the only difference between the two repos was the README file github uses, so I decided to put them together. Here's my workflow now.

All I needed to do was add was update my .git/config file to look include both of the following:

[remote "node"]
        fetch = +refs/heads/*:refs/remotes/origin/*
        url = node@64.30.136.179:repo
[remote "github"]
    fetch = +refs/heads/*:refs/remotes/origin/*
    url = git@github.com:xjamundx/CollabPaintJS.git

Note: At first things were a little bit weird. Both of the repos had changes initally that were not in the other. Fortunately in my example the differences didn't really matter and I just went with one of them. I did this by branching the no.de version into a node branch and then checking it out ontop of master after merging in the ghithub branch. Once I did this everything was fine. Pulling, if you need to do it, is a little different as well, because you might be pulling from two different places:

git pull github master

Overall, for a small project that's being hosted both by github and Joyent, this solution seems to work pretty darn well. Let me know how if it works for you or alternatively, why it's a bad idea!

SQLite and iOS Quirks

This caused me a lot of confusion over the last couple of days.

When selecting you start on 0, using statements that look like this:

self.commentId = sqlite3_column_int(selectstmt, 0);
self.email = [NSString stringWithUTF8String:(char *)sqlite3_column_text(selectStmt, 1)];

But when inserting you start on 1 using similar statements:

sqlite3_bind_text(insertStmt, 1, [self.contactName UTF8String], -1, SQLITE_TRANSIENT);

Another quirk:
It's SQLite, not SQLLite!

It's a bit quirky, but once you get to know how to use SQLite with iOS it becomes really helpful!

P.S.: Here is a tutorial that I found helpful:
http://www.iphonesdkarticles.com/2008/10/sqlite-tutorial-selecting-data.html

Express Pagination Middleware Simplicity

Inspired by watching TJ Hollawaychuk's supremely awesome screencast about using route-specific middleware at http://expressjs.com/screencasts.html, I threw together a simple use case for myself: pagination. While pagination might not be an obvious choice for middleware, it works here because it doesn't need to know anything about the route that's using it, though in this case it may take advantage of some passed in variables if they exist!



Here is a partial using EJS syntax that will display some previous / next arrows wherever you choose to place them. Notice I'm using locals.prev and locals.next here, which works, becuse I'm not sure those variables will even exist. In these cases you access them as properties off of the locals object in the view, which means you don't have to do this if (typeof prev !== "undefined" && prev), which is a little more verbose if you ask me:



I liked how easy the middleware approach was over having specific pagination logic in each route that used pagination. Express is fantastic for letting me send in multiple functions that I can chain together by calling next(). In this example, without using the middleware I would have had to wrap each paginated function in the count() database call, which would give me an un-neccessarily long callback chain. I prefer the simple middleware approach allowed by express.

JS Codings Standards

Here are some JavaScript coding standards/conventions I've been thinking about lately. I put this together pretty quickly, so it may contain some errors. What you think about these? Are there other conventions you support? Please post your feedback and/or corrections and suggestions!