rvm gemset empty mygems ""
Useful when you want to blow away all the gems installed for an rvm version of ruby and start over
rvm gemset empty mygems ""
Useful when you want to blow away all the gems installed for an rvm version of ruby and start over
This actually started with a strange error, when I went to fire up a rails server in an app I’d been working on the night before.
Unable to load application: LoadError: dlopen(/Users/xxx/.rvm/rubies/ruby-2.3.0/lib/ruby/2.3.0/x86_64-darwin14/readline.bundle, 9): Library not loaded: /usr/local/opt/readline/lib/libreadline.6.dylib
Like I mentioned, I’d fired up this app less than 24 hours prior, so this was very strange to me. Some quick googling around suggested that I try to reinstall my ruby version so I ran the command
rvm reinstall 2.3.0
Well that errored out with another strange error
ruby-2.3.0 - #configuring........................................................... ruby-2.3.0 - #post-configuration. ruby-2.3.0 - #compiling........... Error running '__rvm_make -j 1', showing last 15 lines of /Users/xxx/.rvm/log/1476345601_ruby-2.3.0/make.log compiling enc/trans/newline.c compiling ./missing/explicit_bzero.c compiling ./missing/setproctitle.c compiling dmyenc.c linking miniruby dyld: lazy symbol binding failed: Symbol not found: _clock_gettime Referenced from: /Users/xxx/.rvm/src/ruby-2.3.0/./miniruby (which was built for Mac OS X 10.12) Expected in: /usr/lib/libSystem.B.dylib dyld: Symbol not found: _clock_gettime Referenced from: /Users/xxx/.rvm/src/ruby-2.3.0/./miniruby (which was built for Mac OS X 10.12) Expected in: /usr/lib/libSystem.B.dylib make: *** [.rbconfig.time] Trace/BPT trap: 5 ++ return 2 There has been an error while running make. Halting the installation.
You might notice the “(which was built for Mac OS X 10.12)”
which is a bit of a hint at what the problem is.
Because the last line hints the problem is with make, I decided to reinstall make with brew and it dumped the solution to the problem in my lap
brew install make Warning: You have Xcode 8 installed without the CLT; this causes certain builds to fail on OS X El Capitan (10.11). Please install the CLT via: sudo xcode-select --install
Basically I’d upgraded my Xcode version without installing command line tools. Once I ran this command, my rvm reinstall went fine and my readline problems went away.
One of the most interesting management insights I’ve read in a while.
I have a staff member who produces brilliant work but is consistently late every single day. I can’t fire him because it will take months…
Dushka Zapata’s answer: I once hired a woman who did brilliant work but was not a good team player. It’s not that she was destructive or hostile. It was more that, in an industry that typically thrives on collaboration, she was an independent contributor. Another employee was always late. I fel…
the key insight being
People who deliver brilliant work are hard to come by.
It’s our job as managers to make our business environment flexible enough to give every brilliant employee a place to thrive.
I absolutely loved this!
All too often, I find that managers (especially middle managers) are concerned with bringing employees “in line” and getting them to conform to the system, instead of figuring out how to help them do their best work. I imagine following the approach outlined in the article is much easier said than done though.
PS: I bet it would make a very interesting interview question for a management position.
A couple of years ago I worked at a “startup”. I put startup in quotes because it was really a big company pretending to be a small one. Ideas generally flowed from the top down; engineers wound up on the crap end of everything, usually getting overruled and dictated to by PMs and the UX team. Naturally, a lot of the engineers were disillusioned and frustrated with the way things were working.
It was in the middle of this rather cynical time that we got a new PM on the team. He was tasked with improving our onboarding flow and upping conversion numbers, and I was the engineer assigned to work with him on this project.
He had a lot of ideas about how to go about accomplishing this task, and surprisingly, he had actually dug around in the database and pulled some numbers that backed up the direction he was going in (before he came along, very little of what we did was based on data … another source of frustration).
However, on the list of things he wanted to try was something that raised my eyebrow.
“Hey … Pete” (not his real name by the way)
“Am I reading this right? You want us to leave off the subdomain field on signup?”
“So how will we assign subdomains to the client to access and login to their app?”
“We can just randomly generate a unique string and use that, I think”
To provide some context, this was 2013, and the subdomain-login-access pattern where a customer was actually a group of people who needed separate logins to access a particular space within an app, had become a mainstay of web apps all over the techie world. The pattern, popularized by 37 Signals and Basecamp almost 8 years earlier, was simply one of those things that everyone just did.
Being the brilliant undiscovered product genius I thought I was at the time, I knew taking away subdomains from the signup form, would be an unqualified disaster. Users would freak out when they signed up and were redirected to random5tring35.app.com to login. They would panic when they couldn’t remember what url to go to login, they would bombard us with emails, and the complaints would force us to roll back this silly change.
I smirked to myself and got to work on getting this and other changes pushed out on the appointed date. After getting everything built out and QA’ed in staging, we rolled out the change and waited …
I’ll spare you the suspense … nobody complained about the subdomains
I was completely blown away. I simply could not understand it … I mean, subdomains were so crucial to the users, how could they not miss that on the sign up?!?!?
Turns out that subdomains are actually confusing for a lot of users, and us taking that off their plate simplified the sign up process. Users mostly didn’t care what the url looked like, and the ones that did, could request a specific one in the settings. (A few years later, 37 Signals would remove subdomains in the newest versions of their apps, citing the exact rationale that we discovered back then)
I learned 2 things that day
– Humble pie is actually pretty delicious, people should eat it more
– You know nothing Jon Snow. (Seriously. You don’t)
Lots of us walk around spouting truisms without any data to support what we’re talking about, and nobody challenges us, so we keep doing the things we’re doing. In reality, if we poke a couple of these “assumptions” we might find that a lot of them come tumbling down like a Jenga tower.
So, whenever you find yourself not wanting to try something because “everyone knows thats how its done”, take a breath then go ahead and test that assumption.
If you’re correct what do you have to fear?
One of the most common pieces of advice you’ll get as a startup is this: Only hire the best. The quality of the people that work at your company will be one of the biggest factors in your success – or failure.
Perhaps worst of all, if the interview process is predicated on zero doubt, total confidence … maybe this candidate doesn’t feel right because they don’t look like you, dress like you, think like you, speak like you, or come from a similar background as you? Are you accidentally maximizing for hidden bias?
Loved this little gem of a quote hidden inside the following article
“Product management is the art of knowing what to build”
Feels like one of those “Everybody knows that” things from the Geico commercials, but from experience, fewer people than you’d imagine know this.
Stack fallacy has caused many companies to attempt to capture new markets and fail spectacularly. When you see a database company thinking apps are easy, or a VM company thinking big data is easy? – they are suffering from stack fallacy. Stack fallacy is the mistaken belief that it is trivial to build the layer above yours.
There are two ways to fail and the consequences are orders of magnitude apart. Convincing failures allow us to make better decisions in the future. We tend to agonize over the risk of executing something poorly but the much bigger risk is building the wrong thing to begin with.
This is one of my favorite articles about Failure. Written by Andrew Bosworth, almost a year ago now (who I wish would blog more), I still reference it from time to time to remind myself of the lesson in it, which is …
“… if you execute to a level of quality that makes it unlikely that another team, even with more time and effort, could succeed, then yours is a convincing failure. This kind of failure is strategically valuable because we can now eliminate an entire development path from consideration. That means subsequent work is dramatically more likely to succeed. A convincing success is unquestionably the goal of every effort, but a convincing failure should be a close second.”
For me the lesson that lies inside of that nugget doesn’t just apply to Engineering teams, it applies to life.
Lots of times, people will try something, but they’ll do it half heartedly. They might open a business, or try to do standup comedy, or try to make a pro team (that was me), but they don’t give it their all … maybe they want to string some wins together so that they can convince their friends and family that this isn’t some flight of fancy, or maybe they’re afraid that they’ll fail, so if they don’t give it everything they’ve got, it won’t hurt as much when things don’t work out. The problem with this approach is that, just like the article says, you’re still left wondering “what if?” when the opportunity finally fades from view. Which in a lot of ways is worse than not trying at all.
I like reading this article because it reminds me that if I try something in my personal life, that I want to either succeed or I fail … hard. No in between.
I’ve worked on high functioning teams, dysfunctional teams, teams of people who absolutely hated each other, teams where everyone loved each other, and I think I’ve zeroed in on one trait that anyone can look for in a team that will tell you how productive and successful they’re going to be.
Before I do this, I’m going to tell a small sports story.
I played at a pretty high level (amateur) when I was younger, and I played with this one guy (lets call him Dick) who was a brilliant player, with a nasty habit of berating players every time they made a mistake. The thing I noticed was that, sometimes players wouldn’t even make a mistake, they’d just do something that he didn’t think they should do, and he’d immediately get on them. Even worse, sometimes he’d make a mistake and blame someone else.
Miscontrolled a ball? “That was a Shit pass!” he’d yell.
Gave the ball away in a bad area of the field? “Why weren’t you covering that guy?!”
If they ignored him, he’d just keep sniping at them or refuse to pass to them in the game/practice. People didn’t really enjoy playing with this guy at all. He was very good though, so you’d just have to suck it up and try to play with the negativity, which is very tough for younger/developing players who were short on confidence sometimes.
There was another player, (lets call him Wayne), who was actually even better than this guy, but it wasn’t apparent because he did a lot of “dirty work”, unglamorous stuff. He rarely did fancy moves, or tried things that made you go “wow” or look around for an ESPN filming crew. But the guy was steady, technically brilliant, rarely made mistakes (which you start to realize is very hard to do at a high level). If you were in trouble, say surrounded by 2 guys and were about to lose the ball, you could knock the ball to his general direction and it didn’t matter how much pressure he was under or how crap the ball you played to him (look up “hospital pass”), he’d pick it up and find a way out. It was amazing.
Want to know the awesome thing about this guy? he never blamed people for anything!
It was actually kind of weird.
If you made a mistake he’d yell
“Don’t worry about it, just recover, go get it back!”,
If you played a bad pass, he’d say
“Go on … go make sure they make the mistake now … win it back!”.
If you missed the shot, even if it was from 2 yards out with an empty goal, he’d say
“Unlucky. You’ll get the next one!”
He’d constantly yell encouragements …
“Awesome pass John”
“Holy crap look at Ronaldo over here!”
“What a shot … more of that!”
He was everywhere … always looking for a way to help out. If he was better positioned, he’d tell you where to play the pass. If you were hesitant about what to do, he’d yell out
“hit it” … or “touch that off to Mike” … and if you did something different, he’d wouldn’t skip a beat
“ohhh … even better. love that pass!”
then he’d be off to get open for the next guy.
And it was infectious … because he gave you encouragement, you weren’t afraid to make mistakes, and because nobody else was really as good as this guy, you realized that you really couldn’t get on somebody about how crap they were playing, because at some point in the game you’d probably do something pretty stupid. A side effect of this is that everyone started to play for everyone else, because there was no real thing as a “mistake”, everyone just knew to cover for everyone else, even Dick.
So if your man blew by you, somebody else would step up and mark the loose guy, someone else would take his guy, etc etc, and you could just go find an open guy and mark him. Everybody covered for everyone else … it became part of the game, part of the team DNA … and it was a very good team indeed.
I bet by now you know what that trait is in teams I’ve seen that predicts how well they work together. There is no fear of failure.
If you get some bad code into production, you don’t get thrown under the bus, or get walked into a small room with fluorescent lights to get reamed by your boss, instead people swarm around and try to help you, and you do the same when someone else makes a mistake, or misses a deadline. The culture of blame is non-existent, so people fix problems, not the blame. If a deadline was missed, then the team didn’t make a good estimate NOT that Kathy is a shitty engineer.
This is very important to note, because most corporate environments, for whatever reason, seem to embrace a culture of blame. If something goes wrong, everyone wants to know who’s at fault.
“yeah Mike wrote that code 2 weeks ago, we told him not to push it live, but he did it anyway, this is what happens.” … usually followed by a self righteous sigh.
People point fingers quickly, because a culture of blame stems from a culture of fear. Fear of failure, fear of not living up to “high standards” … fear of not being an “A player”. If someone else is to blame for something, then you can still stay secure in your illusion of awesomeness, or maybe you are really fantastic and all your shitty co-workers are just bringing you down, or maybe nobody will listen to your brilliant ideas, so this is what they get.
Now does this mean that you should constantly cover for and try to baby along players who can’t hack it at the level of everyone else? Absolutely not.
On the soccer team I was on, you had to meet a certain (high) standard to make the squad … if you weren’t good enough you simply wouldn’t make the team … simple.You had to have a particular skill level to play, but once you were in, you were one of us, and we’d work our socks off for you, just like we’d work our socks off for Lionel Messi if he chose to join (look him up, he’s only the best player in the world). Its the same way on the great teams I worked with.
Teams that work well together don’t fear failure, High functioning teams that work well together, have amazingly talented and driven people working together for each other who don’t fear failure. They embrace it.
They can do this because they have great team players, men and women who constantly work for each other, push each other, encourage each other, and pick up the slack when somebody falls down or trips up, without even thinking about it. Great teams are stacked with people who say
“What can I do to help”
“I’ve done my job”
You know … quintessential team players. The real tell-tale sign of a great team.
Being that person that makes everybody else great, isn’t that hard. It’s all about being like Wayne, performing at your best, being a pro no matter what and adopting an attitude that there is no real mistake, because you’re going to cover for your teammate, and trusting that they’re going to cover for you.
It really is that simple.
Most people know what it takes to be considered a professional at something, High levels of competence, reliability and exceptionalism … but I want to talk about the last 2% of what it takes to become a consummate professional. It involves how you react to things, specifically shitty situations and people. I’m going to throw out a couple of scenarios to illustrate what I’m talking about
– What do you do when a coworker has just told your boss that your work is awful and you don’t deserve to be at the level that you’re at in the company, and then that coworker comes over to ask for your help with something?
– What do you do when an engineer comes to you (as the lead) with 2 days to launch a big project and tells you the launch date won’t happen because a bug they were trying to fix, quietly (emphasis on quietly, nobody else knew about it) turns out to be much more serious than they thought?
– What do you do when you think a client is clearly stretching out final payment by sending you on wild goose-chases after bugs that aren’t really bugs, or arguing that things you uncovered as issues before the project started are actually your fault; insisting you fix them without charge before you get paid?
– What do you do when your boss takes credit for your work and ideas in the big presentation after weeks of insisting that they would never work, and gets a promotion because of it?
– What do you do when during a friendly debate with a coworker at lunch, they suddenly get personal and heated for no particular reason?
– What do you do when the team lead who shot down your approach for testing and releasing your feature at the last minute, has no problem when someone else attempts the same thing a week later?
– What do you do when you catch your coworker lying about the cause of a bug, trying to throw someone else under the bus for something that was their fault?
– What do you do when a previous company tries to screw you over after you’ve given notice that you’re quitting, examples include messing up your health insurance paperwork, or insisting that you return the company phone even though usually you can pay a fee to keep it upon separation, or badmouthing you through back channels in a small town (workwise)?
– What do you do when you realize a client of your company has a problem with you being gay/black/latino/Middle Eastern/White/a Woman, is trying to get you off their project and your company is refusing to back you up?
– What about if a coworker uses disparaging or condescending language in addressing you on slack or in JIRA tickets?
What I’ve learned over the course of my career is that the times when you need to be at your most professional are the moments when you least feel like it. Responding to underhanded coworkers, dickish bosses, an executive team with no backbone or angry clients when you’re angry and justifiably so, without losing your cool, is one of the hardest things you might ever have to face in your career. Just because you feel justified in responding in kind to a put down or dressing down an underperformer, doesn’t mean that you should. The top professionals I’ve met respond to these kinds of situations by
– Staying completely calm
– Trying to take the emotion out of it
– Trying to resolve the situation without ascribing blame
– Not getting drawn into an emotional exchange
– If the other party is unreasonable or shitty, ending the interaction as quickly as they can or escalating it to someone with more authority (this might mean involving a lawyer if you’re your own boss)
– Avoiding shitty people, but remaining pleasant and polite in all your interactions with them
Taking the high road has the key benefit of getting the crappy situation/person off your mind faster than if you engaged with it. For example, say you respond to a coworker questioning your abilities by getting annoyed and attacking their own abilities. Very quickly the situation escalates into one that requires people to pick sides, or one where everyone is walking on eggshells for several weeks till both of you or one of you is fired or disciplined. Maybe you win, maybe you lose … but what could have been out of your mind after that day, has now turned into a month long ordeal that has lessened everyone’s opinion of you.
High price to pay for “showing them” hunh?
Naturally, this is a very high bar to attain, and in my life, I think I’ve only met 2 people I would consider total pros in this way. My big problem is that its hard for me to remain calm when I’m dealing with people that I think are jerks, its a personal trigger for various reasons, but I think over the last few years I’ve improved very drastically in this area, mainly by coming to the epiphany that just because I may think of someone as a jerk doesn’t necessarily make them one. Like the old saying goes
“If you run into an asshole in the morning, you ran into an asshole.
If you run into assholes all day … you’re the asshole”
Don’t become an asshole by mistake.
Significant new features include &., Array#dig, Hash#dig and the very user-friendly “did you mean” when you get the name of a gem wrong when bundling your gems.
Of course there are performance improvements, but this blog post brilliantly walks through all the new cool stuff in Ruby 2.3.0 … bookmark away!
Also worthy to note that Heroku currently supports Ruby 2.3.0 but is running a preview version, I’d expect an update to the final release before the end of the year
I’ve worked at tech companies/startups for a while now, and as a result I’ve negotiated stock options and salary a bunch of times, and I’m here to tell you that whenever a startup founder or recruiter does hand waving around your stock options, thats not a particularly good sign for you, *especially* if you’re taking a big paycut to work for said company.
Specific examples of this are situations where … say … you want to know how many outstanding shares there are, or how much money you’d make if the company is acquired or goes public. These are completely fair questions, but in the first scenario, there is a trend in the startup space where some founders don’t want to let you know what percentage of shares you’re being given, probably because you’re not getting as much as you think. They’ll say its because further funding rounds will change the number of outstanding shares (aka you’ll get diluted) so its pointless to provide you that number since it may change with a funding round … but I’m a cynic.
In the second, some founders are uncomfortable with employees focusing on money, because they feel folks who do are going to be mercenaries. This is pretty curious specifically because a lot of the time, these same founders are *constantly* talking about money. How much the company is valued at, how much to pay this engineer vs this other engineer, How much money the VC company should give them and why. etc. They’re comfortable with it, they have to be pretty forward about it too, otherwise they won’t get what they need, so its pretty curious that some get uncomfortable when prospective hires do the same thing.
What is interesting is that thoroughly talking through the numbers around your stock options and what it means for you in an exit event, can at once let you know how the founder *thinks* about possible scenarios for the future of their company, whether they’ve even thought about it, and let you know exactly how much risk you should be bearing.
For example, If you’re giving up $40k in salary, and the company exits in 4 years and you only make $160k .. is that worth it? Probably not, right (factoring in inflation)? But what about $350k … $700k? What would make sense for you? Lots of engineers *never* think about this, because we’ve been conditioned to not talk or think about money … because you know … passion.
What if the Founder is talking about IPO’ing at a market cap of $15bn … but all their competitors who have done the same thing are sitting at a $3bn market cap. You’re going to have to ask what makes the company so different, and expect a pretty compelling answer, otherwise you probably shouldn’t make that same bet.
I’ve worked at companies that were completely open with numbers. Stock options, How many? what percentage? What changed with the latest funding round? Quarterly numbers. Valuations. etc etc etc … and I’ve worked at companies that didn’t tell you @#$t. Best companies to work for? The ones that let you know what the hell was going on by far.
Because you didn’t join the startup because you’re a charity, especially if you’re leaving salary on the table at Darth Vader corp. You want to do great work, with great people, but if you’re like most people, you also want to make good money, because money helps make life more comfortable, lets you buy toys you’ve wanted your whole life and gives you lots of interesting options for how proceed with said life.
Its why you should always make sure that your pay package is going to do what you want it to do. Companies that give you this information on a regular basis, allow you to make educated choices instead of blindly following “the mission”.
Places that swat away questions around numbers with “Honestly, the money isn’t that important. We just want people here who are excited/passionate about our mission.” make me nervous, because when they’re hiring that CFO from Google/Facebook/Twitter/Amazon to help them go public, you better believe they wouldn’t dare say something that silly to them, so why would they say that to you?
There are many ways to assess if a candidate is more interested in their compensation package, than working for your startup. Just off the top of my head …
But the truth is, its always about the money … unless you don’t want it to be (you love the founders, the team and the product and could care less about the money … that happens too).
If you’re forgoing salary in hopes of a decent sized payout from your options with LukeSkywalker startup … always remember that most startups fail, no matter what the buzz and the hype looks like at the outset. Of those that don’t, most take a long time to get to a liquidity event, and even fewer still do so in a way that makes a lot of money for its employees.
If you ignore that, you could wind up like these guys …
On Sept. 4, employees of Good Technology, a mobile security start-up in Sunnyvale, Calif., awoke to discover that their company was being sold to BlackBerry, the mobile device and software maker. Some workers immediately began trying to figure out what it meant for Good to abandon its long-anticipated plan to go public – a move that would have potentially turned their shares in the start-up into gold.
I’ve been doing a lot of Zimbra work lately, and since a lot of its is pretty arcane and lightly documented (there is *some* documentation, just not a lot). I’m going to be posting some of my findings on here.
Today’s tip is a basic one. If you want to do ajax requests in the Zimbra extensions you write (zimlets). You’re going to have to turn on a proxy server inside of Zimbra itself. The proxy is actually an Nginx server that takes your requests shoots the “target” url over to the server you’ve specified, then returns the results to your zimlet. (Remember that Zimbra itself runs inside of Jetty, a Java Web Server, proxied to Apache).
Hopefully you have access to the Zimbra Admin screen to streamline the process. If you do all you have to do is navigate to here
Home > Configure > COS > default > Advanced
and just add the domains by hand. You can also do it from the commandline like its specifies in the link above, but I find this to be easier/faster
tldr; mysql2 gem specifies a default wait_timeout of 2147483 seconds that overrides whatever you set on a mysql server
I spent a whole day yesterday trying to debug why updating mysql’s wait_timeout wasn’t working.
Initially, we were having some trouble with a small mysql ec2 instance that was running out of memory because
All pretty silly really. It could have been solved by just getting a bigger box, but there were a bunch of reasons why that wasn’t possible right at that second. And there were similar boxes that had been overlooked but were doing fine, difference was they had fewer connections … about 8k vs our 12k.
In checking out the connections, I realized almost 80% of them were sleeping and had been sleeping for a long time. I figured that setting a low enough wait_timeout (something like 120 seconds or so … average user session limit) on that box would solve the problem. I was worried about “mysql server has gone away” airbrake errors showing up, but I figured it was worth a shot to see if it relieved the memory pressure enough to stop worrying about this particular mysql server dying every few days.
We updated the settings from the mysql console (setting the GLOBAL wait_timeout there), restarted the server, and waited …but the mysql connections just traipsed past the wait_timeout limit.
Initially I thought It’d been done incorrectly, so I manually edited the my.cnf file and restarted the mysql process again, which was painful because even though this was a small box, it had about 11G of mysql in memory that it had to flush.
Again the connections came back up and went past the limit we set.
After investigating for a while, I remembered having encountered a similar problem a few months ago when I tried to do the same thing on a personal project. I had been able to set the wait_timeout on my production server but never got it to work on my local environment.
I raced home and started looking around to see what the difference was between the two. That’s when I realized that I had specified a wait_timeout in the database.yml of my production box but not my dev box. After googling for a bit I finally pieced it all together.
It turns out that the mysql2 gem does this thing …
ConnectionPool uses wait_timeout to decide how long to wait for a connection checkout on a full connection pool, and it defaults to 5 seconds.
mysql2_adapter uses this same
wait_timeout, but passes it directly to mysql, and defaults to a much larger
Things have changed a little from that github issue being filed, but essentially it substitutes a default wait_timeout value of about 25 days (2147483s) of its own as the wait_timeout value (probably the session version of wait_timeout) that the mysql connection uses, basically overriding whatever setting we specify on the server.
By setting a wait_timeout: value in config/database.yml the timeout works as it should. When the connection gets killed, however, you get the infamous “mysql server has gone away” errors.
By specifying an accompanying
option in database.yml, this is fixed, but in a quirky way.
Everytime the connection is reused, say you run an active record command at the console … then run another one, the connection’s timer is reset, BUT after that the connection it uses the wait_timeout setting that you configured from mysql directly.
So if you specified a 10 second timeout in your database.yml, and 15 seconds in your my.cnf file, then you logged into the console and ran an Active Record command.
Mysql would spawn a connection that would sleep after it was done for 10 seconds before being closed by mysql.
If before the end of that 10 seconds you ran another command, it would execute it and then start to sleep again, but now (and every time after) it would sleep for the 15 seconds specified in my.cnf before being killed by mysql.
Hope that piece of trivia brightens up your day 😀
I came home from a week long company hackathon to see my newrelic error reporting going crazy …
My free bugsnag account had already maxed out of the 2000 error allocation I had for the entire month (I’ll usually get 100 in a month … maybe).
I immediately knew from the exception in new relic that something was wrong with Sphinx, and I figured it was pretty bad because on each page load of news pages, I hit sphinx to give me a list of related news stories, which meant that the crawlers that hit my site every second of the day could be generating loads of exceptions.
However, when I went to pull up one of my news pages … it loaded just fine …
I hit the search page, and got a 500 error. Weird as that may be, I’d encountered that before. I immediately logged into my vps and ran
at the console and was ready to drop my mic and moonwalk back to my sofa to catch up on “The Americans”.
Then I looked at my new relic account and noticed that the same error was still coming in 10 minutes after it should have been fixed.
I tried hitting the urls specified in the errors and got the 500 error. I realized that some urls were exhibiting the problem while others weren’t.
I’d never seen that before :\
My google-fu quickly turned up this non-upvoted gem that helped me fix the issue. Basically you just have to rotate your sphinx index because that error means parts of it have gone “stale”.
the command I used to accomplish this was
`/usr/bin/indexer –rotate xx_core –config /path/to/your/sphinx/production.sphinx.conf`
your indexer command might be located somewhere different though; to locate mine I just used the `whereis indexer` command
You’ll want to change xx_core to match the name of your index (which you can find by looking in your sphinx.conf file)
Hope this helps you out!