Using rails and having trouble getting Mysql’s wait_timeout to work? read this.

tldr; mysql2 gem specifies a default wait_timeout of 2147483 seconds that overrides whatever you set on a mysql server

I spent a whole day yesterday trying to debug why updating mysql’s wait_timeout wasn’t working.

Initially, we were having some trouble with a small mysql ec2 instance that was running out of memory because

  1. it had less ram than other boxes doing the same thing (we’d forgotten to upgrade it)
  2. it would spawn too many connections (12k or so) which take up memory and because there was no swap specified, when mysql grew too big, it would get killed by the os.

All pretty silly really. It could have been solved by just getting a bigger box, but there were a bunch of reasons why that wasn’t possible right at that second. And there were similar boxes that had been overlooked but were doing fine, difference was they had fewer connections … about 8k vs our 12k.

In checking out the connections, I realized almost 80% of them were sleeping and had been sleeping for a long time. I figured that setting a low enough wait_timeout (something like 120 seconds or so … average user session limit) on that box would solve the problem. I was worried about “mysql server has gone away” airbrake errors showing up, but I figured it was worth a shot to see if it relieved the memory pressure enough to stop worrying about this particular mysql server dying every few days.

We updated the settings from the mysql console (setting the GLOBAL wait_timeout there), restarted the server, and waited …but the mysql connections just traipsed past the wait_timeout limit.


Initially I thought It’d been done incorrectly, so I manually edited the my.cnf file and restarted the mysql process again, which was painful because even though this was a small box, it had about 11G of mysql in memory that it had to flush.

Again the connections came back up and went past the limit we set.

After investigating for a while, I remembered having encountered a similar problem a few months ago when I tried to do the same thing on a personal project. I had been able to set the wait_timeout on my production server but never got it to work on my local environment.

I raced home and started looking around to see what the difference was between the two. That’s when I realized that I had specified a wait_timeout in the database.yml of my production box but not my dev box. After googling for a bit I finally pieced it all together.

It turns out that the mysql2 gem does this thing …

ConnectionPool uses wait_timeout to decide how long to wait for a connection checkout on a full connection pool, and it defaults to 5 seconds.

mysql2_adapter uses this same wait_timeout , but passes it directly to mysql, and defaults to a much larger 2592000!!

Things have changed a little from that github issue being filed, but essentially it substitutes a default wait_timeout value of about 25 days (2147483s) of its own as the wait_timeout value (probably the session version of wait_timeout) that the mysql connection uses, basically overriding whatever setting we specify on the server.

By setting a wait_timeout: value in config/database.yml the timeout works as it should. When the connection gets killed, however, you get the infamous “mysql server has gone away” errors.

By specifying an accompanying
reconnect: true
option in database.yml, this is fixed, but in a quirky way.

Everytime the connection is reused, say you run an active record command at the console … then run another one, the connection’s timer is reset, BUT after that the connection it uses the wait_timeout setting that you configured from mysql directly.

So if you specified a 10 second timeout in your database.yml, and 15 seconds in your my.cnf file, then you logged into the console and ran an Active Record command.

Mysql would spawn a connection that would sleep after it was done for 10 seconds before being closed by mysql.

If before the end of that 10 seconds you ran another command, it would execute it and then start to sleep again, but now (and every time after) it would sleep for the 15 seconds specified in my.cnf before being killed by mysql.

Hope that piece of trivia brightens up your day :D

ActionView::Template::Error: No response from searchd (status: , version: )

I came home from a week long company hackathon to see my newrelic error reporting going crazy …

sphinx error driving new relic CRAZY

sphinx error driving new relic CRAZY

My free bugsnag account had already maxed out of the 2000 error allocation I had for the entire month (I’ll usually get 100 in a month … maybe).

I immediately knew from the exception in new relic that something was wrong with Sphinx, and I figured it was pretty bad because on each page load of news pages, I hit sphinx to give me a list of related news stories, which meant that the crawlers that hit my site every second of the day could be generating loads of exceptions.

However, when I went to pull up one of my news pages … it loaded just fine …


I hit the search page, and got a 500 error. Weird as that may be, I’d encountered that before. I immediately logged into my vps and ran

`rake ts:restart`

at the console and was ready to drop my mic and moonwalk back to my sofa to catch up on “The Americans”.

Then I looked at my new relic account and noticed that the same error was still coming in 10 minutes after it should have been fixed.


I tried hitting the urls specified in the errors and got the 500 error. I realized that some urls were exhibiting the problem while others weren’t.

I’d never seen that before :\

My google-fu quickly turned up this non-upvoted gem that helped me fix the issue. Basically you just have to rotate your sphinx index because that error means parts of it have gone “stale”.

the command I used to accomplish this was

`/usr/bin/indexer –rotate xx_core –config /path/to/your/sphinx/production.sphinx.conf`

your indexer command might be located somewhere different though; to locate mine I just used the `whereis indexer` command

You’ll want to change xx_core to match the name of your index (which you can find by looking in your sphinx.conf file)

Hope this helps you out!

Is Ruby pass-by-reference or pass-by-value?

I was confused by Ruby’s variable assignment paradigm for a while, and even after reading this Stackoverflow thread and checking out out other threads about the same thing I was no closer to understanding it clearly. And in the words of the great Albert Einstein …

“if you can’t explain it simply you don’t understand it well enough”

I eventually figured it out by reading page 53-54 of what I think is the best Ruby book on the market right now, turns out its actually pretty simple to understand.

Ruby ALWAYS passes references to objects on assignment, BUT some types in ruby are stored in variables as immediate values because they are immutable.

For example (you can try these out in irb)

me = we = “we”
me #=> “we”
we #=> “we”
me #=> “WE”
we #=> “WE”

me = we = [1, 2, 3]
we #=> [1, 2, 3]
me #=> [1, 2, 3]
we << 4
we #=> [1, 2, 3, 4]
me #=> [1, 2, 3, 4]

Ruby does this for all its types, but with primitives that are immutable, namely integers, symbols, nil, and the booleans true and false, it takes a shortcut and actually stores the value IN the variable itself, because … what are you going to do? change an immutable object? :)

Pretty clever when you think about it IMO.

Controlling Elasticsearch memory usage on os x

By default, Elasticsearch will use between 256MB and 1GB of memory. On my dev environment, memory is a premium and I don’t really have a lot of data to index and play with so I’d prefer this to be closer to 256m

To change this, navigate to

(where x.xx.x is the Elasticsarch version number)
open up

then replace the memory size in the following lines

if [ "x$ES_MIN_MEM" = "x" ]; then
if [ "x$ES_MAX_MEM" = "x" ]; then

It is recommended that you set both ES_MIN_MEM and ES_MAX_MEM to the max size you want so that Elasticsearch doesn’t have to pause to resize memory, which can affect performance.

from the comments in that file …

min and max heap sizes should be set to the same value to avoid
stop-the-world GC pauses during resize, and so that we can lock the
heap in memory on startup to prevent any of it from being swapped

I upgraded to Ruby 2.1.0 and all I got was this lousy T-shirt …

Ruby 2.1.0 was released about 2 weeks ago, and after seeing a notable speed performance improvement with my Rails app when going from 1.9.3 to 2.0.0 I was excited to see if the same would happen with Ruby 2.1.0.

I quickly ran another set of very casual tests, mainly running test suites and noting startup times of rails commands and rake tasks at the command line. What I saw was a 20% improvement in those tasks. Nothing to sneeze at …

Excited, I rushed to upgrade my tiny Rails app to ruby 2.1.0 and after doing it I tracked the difference in New Relic

This was my result (deployment happened about 18:00 on December 25th, app is setup to run with one puma worker and 8-16 threads proxied to nginx)

New Relic graph after upgrading to Ruby 2.1.0 (18:00 on the graph)As you can see, there wasn’t much of an improvement in performance at all. In fact here is the graph after a full day of running in the wild.

New Relic graph of my app after a full day of running with Ruby 2.1.0


So while you might get better startup times on Rails tasks, you probably won’t see too much of a speed boost on your servers.

Improved GC

What I found interesting, though, is the improved garbage collection. The brown part of the graph (GC execution) almost completely disappears in ruby 2.1.0. And its no wonder, a lot of work went into improving Garbage collection in ruby 2.1.0

The garbage collector in Ruby 2.1 implements a form of generational garbage collection, with Ruby calling their implementation RGenGC (Restricted Generational Garbage Collection).  This replaces the “Mark & Sweep” implementation used in previous versions of Ruby

Using RGenGC provides high compatibility with existing extensions while still bringing performance improvements.  Popular objects Array, String, Hash, Object, and Numeric are Write-Barrier protected, thus able to take advantage of the RGenGC system

The moral of the story? Ruby 2.1.0 has a small performance boost over Ruby 2.0 and upgrading to it should give you a bit of a boost with your Rails app, more if its a pretty big app with non-trivial time spent doing GC.

PS: Brian Hempel has created an awesome site that benchmarks ruby versions using Rails. His results are in line with what I found
PS2: Sorry there wasn’t actually a T-shirt, I just liked the title

fixing Gitbox on Apple OS X Mavericks


As soon as Mavericks was released I rushed to install it, and for the most part, the upgrade went smoothly … with one exception. Gitbox wouldn’t start.

I upgraded XCode, uninstalled, and reinstalled it, then restarted OS X multiple times, but each time I clicked on the icon in the dock, it would light up but the familiar gitbox window with all my repos would not appear.

For a while I made do with Tower (which I started to like after a while), but eventually I found this little nugget of a tweet

Essentially, to get gitbox working again, you need to open up terminal and just type

defaults delete com.oleganza.gitbox

Hit enter and that’s it. When you open up Gitbox again, it will start up, but all your repos will be gone. Just re-add them.


Product idea: iphone music auto resume when changing music sources


Wouldn’t it be cool if when you disconnected your headphones from your iphone and plugged it into an aux jack from your car, the music just continued playing?

I face this issue each time I’m coming from the gym, in the zone listening to a song, then I get into my car and have to change the connections. I absolutely HATE having to bring up the music app and hit play to keep going. It would be amazing if for 5/10 seconds after you disconnect the headphone jack, the iphone just auto resumes playing whatever was playing before, if you plug it into something else. The interesting thing is that if you connect to a bluetooth device it already does this by default (continues playing from where you left off), this would just be a more explicit extension of that user experience pattern.

how to know if your 2011 macbook pro can run with 16GB of ram

2012 and newer macbook pros can run with 16GB of RAM, but 2010 and 2011 models are not supposed to be able to do the same … allegedly.

Turns out that even though the apple tool that tells you the maximum amount of memory you can run with the 2011 models is 8GB, you might still be able to actually upgrade to 16GB, if your cpu is an i7 27xxQM or 28xxQM processor.

Drop into your command line and run this command
> sysctl -n machdep.cpu.brand_string
Intel(R) Core(TM) i7-2760QM CPU @ 2.40GHz

As you can see, in my case, the cpu is the correct one for an upgrade.

The RAM in these is usually 1333 MHz PC3-10600 DDR3 SDRAM but for the upgrade you’ll want 1600 MHz PC3-12800 DDR3 SDRAM

PS: Of the 2010 models, only the 13″ models can take 16GB of RAM, my source for all this information was the macbook pro wikipedia page.


Does your product fit into my life and make it awesome?

I absolutely love this Apple ad …

Whenever it comes on, I completely stop whatever it is I’m doing and watch it.

There is another ad I love almost as much, and its this one by google

What I love about these ads is how they tell you a story.

One in which the product is just a supporting actor.

A story of how their technology fits into your life and makes it awesome.

You see, some people can relate to wanting to have a cooler phone than the other person.

Some people can relate to having a more expensive car or getting a table at that impossible-to-get-reservations restaurant.

Yeah, people get off on having stuff that no-one else has.

But that is a small subset of human aspiration.

The things people really want are usually much simpler than that.

People want to see things they’ve never seen before, go places they’ve never been, hang out with people they like, eat awesome food, have someone think they’re attractive … even sexy … then share all those with people they know

People want to make their friend laugh, have them think they’re a riot.

People really just want simple things things most of the time.

So you can actually make your product huge by showing people how it helps them do the little things.

How you seamlessly fit into their life and make it awesome.

If you can do that … You’ve won.

Thats how facebook got big.

Thats how instagram got big.

Thats how the ipad got big.

They are personal awesomeness force multipliers.

As an example, shows you breathtakingly beautiful places that it has listed when you arrive at the site.

They fade in and out in the background.

Pazin, Croatia. Barcelona, Spain. Copenhagen, Denmark.

Beautiful homes, in beautiful places. for $80 a night!

It makes your imagination run wild. You start to wonder …

“A plane ticket would be … ,”

“If I stayed 2 nights I could do this for less than …”

You start to create your own story in your head. because you can’t help it, you’re human.

And without even seeing a single ad, you’re hooked.

Because airbnb can fit into your life and make it way awesome.

So the question if you’re a product maker is … Whats your story?

How does your thing seamlessly fit into my life and make it awesome?

Relevant: Want to Increase a Product’s Value by 2,706%? Give It a Story

Student loan debt, millenials and the sharing economy


RE: Student Debt Slows Growth as Young Spend Less — NYTimes

Earlier this year there was a lot of hand wringing about why younger kids aren’t buying cars, or houses etc, and the consensus seemed to be that the financial turmoil of 2007/8, and its cavalcade of side effects was the reason, that as the economy improved, things would go back to normal. The writer, wondered, however …

“But what if these assumptions are simply wrong? What if Millennials’ aversion to car-buying isn’t a temporary side effect of the recession, but part of a permanent generational shift in tastes and spending habits? It’s a question that applies not only to cars, but to several other traditional categories of big spending—most notably, housing. And its answer has large implications for the future shape of the economy—and for the speed of recovery.”

It seems like that is exactly the case, and the big catalyst for this change is crushing Student loan debt, coupled with the credit crunch, and slow job market from the near-financial- apocalypse of 2007/8

“It is a new thing, a big social experiment that we’ve accidentally decided to engage in,” said Kevin Carey, the director of the Education Policy Program at the New America Foundation, a research group based in Washington. “Let’s send a whole class of people out into their professional lives with a negative net worth. Not starting at zero, but starting at a minus that is often measured in the tens of thousands of dollars. Those minus signs have psychological impact, I suspect. They might have a dollars-and-cents impact in what you can afford, too.”

What trips me out is how because of this … new ways of thinking about ownership have been spawned that have the potential to reshape our world going forward.

The typical new car costs $30,000 and sits in a garage or parking spot for 23 hours a day. Zipcar gives drivers access to cars they don’t have to own. Car ownership, meanwhile, has slipped down the hierarchy of status goods for many young adults. “Zipcar conducted a survey of Millennials,” Mark Norman, the company’s president and chief operating officer, told us. “And this generation said, ‘We don’t care about owning a car.’ Cars used to be what people aspired to own. Now it’s the smartphone.”


Millennials, of course, are sharing more than transportation: they’re also sharing living quarters, albeit begrudgingly, and with less gee-whiz technology involved. According to Harvard University’s Joint Center for Housing Studies, between 2006 and 2011, the homeownership rate among adults younger than 35 fell by 12 percent, and nearly 2 million more of them—the equivalent of Houston’s population—were living with their parents, as a result of the recession. The ownership society has been overrun by renters and squatters.

Car2go, zipcar, airbnb, uber, Spotify, rdio … millenials aren’t buying to own any more, they’re renting/sharing and the ramifications of this could be massive.

In the end, its really just amazing how the world seems to undergo these seismic changes while nobody is paying attention, and then, just like that … everything changes.

Also Read: The Cheapest Generation 

Showing better highlighted search result fragments with Elasticsearch

Elasticsearch has a pretty awesome highlighting feature, but it comes with a major deficiency. When it truncates your document/string, it gives you no indication that it has done so.

Take a look at this screen shot

As you can see the text (bolded behind the dropdown of results) is truncated in the results in the dropdown itself, but there’s no indication that is what has happened.

Doesn’t seem like a big deal, but for the perfectionists and craftsmen out there, this has to make you itch right? How is someone to know that there is more to that fragment of text that what they’re seeing?

Well, heres some ruby code to the rescue. Throw it in a helper and call it in your view or wherever

  def ellipses_for_highlights(params_highlight, params_original)
    # have to do this because highlighted stuff from ES has a trailing space for whatever reason
    stripped_highlighted_item = strip_tags(params_highlight).rstrip
    # if the beginning of the highlighted text doesn't match the original it has been clipped
    tmp = params_original =~ /#{stripped_highlighted_item}/
    front_ellipsis = tmp != 0
    # if the last 10 characters of the highlighted text don't match the original, same deal
    back_ellipsis = last_string_chars(stripped_highlighted_item, 10) != last_string_chars(params_original, 10)
    highlighted_item = front_ellipsis ? "... " + params_highlight : params_highlight
    highlighted_item = back_ellipsis ? highlighted_item + " ..." : highlighted_item

Link to github gist here

to use this, just pass in the highlighted string from elasticsearch and the original string for comparison.
so something like this


and you’ll get something like this

It will only truncate on the front or back of the string if elasticsearch only truncated at that spot, in addition to truncating on both ends if it realizes that elasticsearch did too. Better, right?

Couple of things to note.
- This will only work cleanly if you have  :term_vector set to “with_postions_offsets” in your mapping. This enables elasticsearch break the fragment on words vs truncating in the middle of a word. If you have it turned off (i.e you’re just using the plain highlighter), you’ll get something that looks more like this (notice how the truncation is happening in the middle of words)

- Also keep in mind that because of the behavior explained above when using term_vectors in your highlighting, the fragment_size will not match the number you specify exactly, makes sense (because it has to break on a word which can have be any number of characters in it), but its not described anywhere

Adding Autocomplete using elasticsearch

A commonly-requested feature in search applications is autocomplete or search suggestions. The basic idea is to give users instant feedback as they’re typing. Implementations of this feature can vary — sometimes the suggestions are based on popular queries (e.g., Google’s Autocomplete), other times they may be previews of results (e.g., Google Instant). The suggestions may be relatively simple, or they can be extremely complex, taking into account things like the user’s search history, generally popular queries, top results, trending topics, spelling mistakes, etc. Building the latter can consume the resources of a company the size of Google, but it’s relatively easy to add simple results-based autocomplete to an existing elasticsearch search application.


How I became a scientist (quote)

My mother made me a scientist without ever intending to.
Every other Jewish mother in Brooklyn would ask her child after school, “So? Did you learn anything today?”
But not my mother …
“Izzy,” she would say, “did you ask a good question today?”
That difference – asking good questions – made me become a scientist.
— Isidor Isaac Rabi, Nobel laureate

Found this gem on the 37 Signals blog