php-fpm 503 timeout error triggered by interrupted client connection

My team and I came across a puzzling 503 error from our httpd / php-fpm application, and the answer turned out to be so different from what we expected that I felt this was worth sharing. What initially looked like a server-side problem turned out to be something else entirely.

We were doing some basic modernization of one of our older applications. It was a simple API endpoint, written in php with an httpd front end. The application was several years old and was running directly on the linux OS, with php running directly inside httpd as module.

Definitely time for its 75,000 mile service. We were embedding it into docker containers to make it more portable, upgrading the php to the latest version, and switching to the more modern php-fpm arrangement. Instead of running php directly inside httpd, the web server would now use fastcgi proxy over to a separate php-fpm container to do the real work.

Everything was looking good through our testing, so finally the cutover came. We re-routed our traffic to the new containerized applications, and everything looked beautiful. Application was working like a charm, throughput was good… job well done.

Being an obsessive sort, I kept checking back in on the systems over the course of the day, and after a couple of hours, I saw we had recorded a half dozen 503 errors amongst tens of thousands of successful requests. Accompanying the 503 message in the access log was this message in the error log (ip addresses and urls obfuscated):

[proxy_fcgi:error] [pid 111] (70007)The timeout specified has expired: [client] AH01075: Error dispatching request to :, referer:

Uh-oh, that’s a proxy timeout with the dreaded 503 response code. You typically see these when something is going way too slow on your server, and your proxy (load balancer, httpd, nginx, etc) eventually gives up waiting around and sends a 503 error back to the client. They are hard to debug because usually it is some sort of data or production scale issue that doesn’t show up in normal testing, and they can come and go. I’ve basically been trained that when I see a 503 error, it’s time to get out my performance tuning toolbox and get to serious work.

However, nothing about this presented like a performance tuning situation. The API calls were super fast, taking just a fraction of a second, and the system resources looked healthy. What could it possibly be doing that would take a 100 millisecond request and trigger a proxy timeout that was set to one minute?

Digging further, the problem became more mysterious. In the PHP logs, we found some application errors that would match up with the recording of the 503 event, but the errors suggested that the application had received a post request without any actual data being posted. That was a useful clue, but it didn’t make any sense – an empty request should be an immediate failure, not the sort of thing that would cause a timeout error.

We tried simulating a POST request without a body using Postman, and we saw what we expected… the same application errors, but a response within a few tens of milliseconds and a 200 from the server. Nothing matching the 503.

Continuing to look further, more strange things popped up. Looking at the httpd log entry for when the request was received versus when it actually responded, we saw what looked like a 20 second drift. httpd was triggering the 503 timeout after 20 seconds, not the 60 seconds it was configured for. Furthermore, those application errors appeared right at the end of the 20 seconds…not the beginning.

As far as we could tell, it seemed like httpd was hanging out for 20 seconds, and then finally telling php-fpm to go do its thing. php-fpm responded that it was missing post data, but httpd had already given up and sent the 503 to the client.

Now a possible explanation was starting to form. Our clients have users all over the world, some of whom have some very poor internet connections. Perhaps what we were seeing was a request starting from a client but somehow never finishing. If httpd then decided to give up after 20 seconds and pass php-fpm whatever it had, this would all start to make sense.

It felt like a good explanation, but I wanted to be sure. Under the covers, the http protocol is just a fancy telnet session. If I could figure out the right series of commands, maybe I could trigger it.

It took some monkeying around, but I finally stumbled on the right combination (IPs and urls again obfuscated):

computer:~ jeremy$ telnet 80
Connected to
Escape character is '^]'.

POST /app/apiendpoint HTTP/1.1
Content-Type: application/x-www-form-urlencoded
Content-Length: 100

In the block above, I am opening up a connection on port 80 to the system using old fashioned telnet and then manually entering the http protocol commands. I first tell it I’m going to do a POST operation, then a few headers, and then the data.

Most significantly, I am giving it a content-length of 100. I am telling the server… “get ready, I’m going to send you 100 bytes of data.” I then send it “helloworld” and go silent. The server has gotten 10 bytes of data, and it is now waiting expectantly for the other 90 bytes.

But I have gone quiet. I haven’t closed the connection, butI haven’t sent the promised data, and the server is waiting. After 20 seconds, the server decides that it isn’t going to wait around any longer and severs the connection.

And then, it response with this:

HTTP/1.1 503 Service Unavailable
Server: Apache
Content-Type: text/html; charset=iso-8859-1
Strict-Transport-Security: max-age=16070400; preload
Date: Wed, 11 Jan 2023 18:02:39 GMT
Connection: close
Content-Length: 299

<title>503 Service Unavailable</title>
<h1>Service Unavailable</h1>
<p>The server is temporarily unable to service your
request due to maintenance downtime or capacity
problems. Please try again later.</p>
Connection closed by foreign host.

And sure enough, we get the 503 in the access log and the php logs generate the application errors about the missing data.

So, this wasn’t a problem in our application to fix, this was a client connection error.

For me it was eye opening. 503 has always meant “problem on the server” to me. It turns out it might also be triggered by a client with a bad connection!

I know there are lots of posts on 503 errors out there, but I’m hoping to add mine to the mix in case maybe it will point somebody in the right direction.

Posted in Uncategorized | Leave a comment

Shortcut: Simple rubber strips to make a phone easier to hold without the bulk of a case

A few weeks ago, my iPhone pulled a disappearing act worthy of Houdini.

I was taking an early 6:30 am shuttle flight from Boston to La Guardia. As the plane taxied away from the gate, I checked for any last email messages and then put my phone into airplane mode as instructed by the flight attendants. I then slipped my phone into my pocket, put on my noise canceling headphones, and settled in for a short nap.

I woke up about 40 minutes later when we touched down. As always I reached into my pocket for my phone and discovered to my surprise it wasn’t there! It was a shiny jet black iPhone 7, and from time to time it has slipped out of my pocket, so I wasn’t concerned. I felt around the seat, but it wasn’t there either.

I figured it must have slipped down between the seat and the cabin wall to the floor, so I started searching under the seat.  No phone.

I then asked the woman behind me if it had slid back to her bag. “Oh, I heard something drop earlier! Let me look,” she said. But no phone.

I then asked the person in front me, but again, no phone.

I knew the phone had to be there somewhere, so I let the other passengers deplane and started a search. The black phone should have been easy to spot against the purple/blue rug, but it was nowhere. With everyone cleared out, it was easy to see under the seats for rows in all directions, but no luck.

The four flight attendants immediately pitched in to help, but it was nowhere.  We looked in seat back pockets, and we removed the cushions from the seats themselves. The captain even went to ask at the gate whether someone had turned it in, but it was nowhere.  There were no crevices for it to have slipped into. It had just vanished into thin air.

Naturally, they offered to try calling the phone for me. When I explained to them that it was in airplane mode, they gave me a look of surprise, as though they didn’t actually expect anyone to follow those instructions.

To this day, I am baffled about what happened to it. The most likely explanation at this point is that someone picked it up and pocketed it, but I still have a hard time believing this. It was a commuter flight from Boston to New York, and the iPhones can’t be wiped if they aren’t unlocked by the owner. What would be the point of stealing it?

Sadly, I had no choice but to walked into the Apple Store in Grand Central station and buy a new one. I couldn’t function traveling to New York for the day without a cell phone. Luckily, Apple has a great program for people in my situation – if you give them $800, they give you brand new phone! I opted for a the bright shiny red ones Apple had released just a few weeks earlier.

By lunch time, I had restored from the previous night’s iCloud backup and was back in business. Well, minus $800 and with a phone that had absolutely no new features beyond being the color red.

It was time, however, to reconsider one of my strongly held opinions – I don’t use a case for my phone. I really hate the bulk that cases add. When people complain that Apple doesn’t use a bigger battery and wonder who really cares that their phone is thin, the answer is me.  I really want it to be as thin as possible. I had a case for a while on my old iPhone 6 Plus, but I hated it – especially when it wore out my jeans.  I had switched to using a decal to add a bit more friction, but with the new styling on the 7, I wanted to show it off.

As much as I have liked the case-free look, it has had its drawbacks. A few months ago on a trip to Atlanta, I put my wallet, phone, keys, and bluetooth headset into a basket to go through the x-ray machine. When I got to the other side of the metal detector, I found the wallet and phone in the basket, but they keys and headset were missing. I slipped them in my pocket and searched around to see what had happened to my keys but couldn’t find them.

Then, another basked came out of the x-ray machine, which had my wallet, phone, keys and bluetooth headset. I had picked up someone else’s identical looking wallet and phone and nearly walked off with them!

And of course, there was this fresh incident of losing the whole phone and paying $800 to replace it.  I liked the look, but I just hated how slippery the phone was. A few times the phone nearly slipped out of my hand. I wanted it to just be a little more “grippy”.

For about a week, I tried a case.  It was a bumper case that still showed off the back of the phone, and the rubber edge made it easy to grab. My wife took one look at it and said, “You are going to hate that.”  She has a case on her iPhone and prefers it that way, but she knows that I prefer the minimalist look.

And she was right. I hated it. I found the lip around the front the phone that is designed to prevent scratches to be uncomfortable to use, particularly when I needed to slide my fingers to the edge of the phone. Really what I wanted was just the rubber grip without the bulk.

After exhaustively searching for other cases, I started to look beyond cases, and I stumbled on just what I was looking for – Strip!

These are simple rubber strips you can apply to the edges and/or back of your phone to give it some more grip.  They have two flavors – either a smooth rubber strip or a rough version with small bumps for even more friction.

I wasn’t sure which one would be best, so I ordered a combo pack that let me try both. With the $800 bill still fresh on my mind, I opted for the rough version for maximum grip.

The strips come in many different sizes allowing you to place them wherever you feel they would work best. I opted for medium ones on the size, and a pair of long ones on the back.


Medium strips on the sides give extra grip when holding.


A pair of longer strips run along the back

So far, they are giving me exactly what I wanted – more friction, without the bulk. I’ve been using them for a week, and they haven’t had issues coming of while rubbing against my pocket.

The real test will be when I take my next trip and we see if they slip out of my pocket again.

When I lost my phone in New York, my coworkers were quite sympathetic to my plight. However, once I was up and running, they proceeded to tease me mercilessly – which I probably deserve.

In the meantime, I think I am also done putting my phone into airplane mode when I fly.

Posted in Uncategorized | Tagged , , , | 2 Comments

Shortcut: Overcome cell network congestion by dropping to 4G

Ever been in some crowded area, like an airport, and tried to pull up a web page on your phone? Your connection strength looks fine with three or four bars, the LTE symbol shows up, but the pages just hang…

I recently found myself in just this predicament while waiting to board the shuttle from Boston to NYC. I had my laptop open, and I was studying a large 100MB trace file somebody had sent me. After looking at it, I knew I needed to send it along to another team to assist, and I really didn’t want to have to wait until I arrived at my destination. If I could just upload it to an internal file share before my flight, they would be able to grab it and perhaps get me their analysis when I landed.

But, when I tried to upload it, I found my network connection was stuck in molasses.  The signal looked strong, but pages just were not loading. The problem wasn’t that I was too far from a tower. It was that thousands of people around me were trying to use that same tower, and we were in a traffic jam.

I’ve found a trick that often works in these situations.  To return to the traffic jam analogy, everyone who has driven in the city knows that during rush hour, the high speed highways aren’t always the best choice. Taking some backroads that have lower, 35 mph speed limits can often be much faster than sitting in a standstill on a highway that is supposed to go 65 mph but too may people are using it.

What is the backroad for your LTE connection? 3G!  (or 4G, as AT&T has branded it).

Your phone always defaults to LTE, but you can force it down to 3G if needed.  Here are the steps for an iPhone:

Go to Settings > Cellular, and find the “Cellular Data Options”:


From here, change the settings for “Enable LTE”:


Change the setting from “Voice & Data” to “Off”, and LTE will be disabled on your phone.


Within a few seconds, your phone will switch over to the 3G (or 4G under AT&T’s branding) network.

For me, this did the trick. Web pages started to move, and my 100MB upload started to progress.  I left my laptop on during the boarding process, and when we got ready to pull away from the gate 15 minutes later, my upload had finished and another team was looking at the file.

I’ve used this trick multiple times successfully.  It seems to work best in airports, particularly, after you have landed and are sitting on the taxi way, waiting for a gate to open up.

It’s not 100% effective, though. For example, the Amtrak waiting area in Penn Station in NYC never moves any data on either LTE or 3G/4G, regardless of signal strength.

Still, it works often enough that it is a useful tool to keep in your travel toolbox.

Let me know if it works for you!


Posted in Uncategorized | Tagged , , , , | 1 Comment

If you travel frequently, take a look at Arro app for taxis

I find myself traveling down to New York from my home base of Boston at least once a month.  While I generally aim for taking the subway and Amtrak, there are a lot of times I find myself in a taxi. Outside of rush hour, it’s the fastest way to get to and from Boston’s South Station. And, while I have become quite familiar with navigating Manhattan by subway, after a late evening dinner I generally hop in a cab back to my hotel.

And when I say taxi, many times I really mean Uber. Like many people, I fell in love with Uber’s convenience. I knew that *my* driver was coming to pick *me* up so I didn’t have to wander around looking for a cab. Best of all, once the Uber ride was done, I just got out. No messing with credit cards or tips.

My favorite part of Uber is actually the electronic receipts. I track all of my travel expenses with Expensify, which has a nifty feature that allows you to just email them your receipts, and they will catalog them for you, just like TripIt does.

But as much as I love Uber, there is a lot to be said for taking good old-fashioned taxis. Manhattan is full of them, and when they are whizzing by you, it often feels silly to pull out your phone, call an Uber, and then wait several minutes for it to arrive. Same thing in Boston… I get off the train at 9:45 pm, and there is a line of taxis.  Much easier to just hop in one and go home.

Unfortunately, it means extra time at the end… I get to my destination, and just when I want to get out and go to bed, I find myself needing to mess with the cab interface, punching in the tip amount, then swiping my credit card, and finally getting a receipt, which I will need to later photograph and enter into Expensify to track.

And then I found Arro.

About half of the taxis I ride in both NYC and Boston were constantly talking about the benefits of the Arro app on the annoying little TV they have. It claimed to be just as good as Uber… you could use it to call a cab, and then when you got out, they would email you a receipt. Even better, if you hopped into a random cab that was part of Arro’s network, you could just punch in the code and Arro would handle seamless payment for the trip.

One night, as I was riding in a cab home, I decided to give it a try out of curiosity.  I downloaded the app, registered myself, and then punched in my credit card. I then punched in the cab’s paring code, and voila. The taxi recognized me, and that was it. Once I got to my destination, I just got out. It added the 20% tip for me automatically and sent me a receipt. Just as easy as Uber.

Currently, Arro operates in New York, Boston, and Chicago, which is perfect for me.  They have plans to expand soon to San Francisco and Washington, D.C.  I’d say about half the taxis I get into use Arro.

When I get in the taxi, I look at the tv screen and check to see if it is an Arro taxi.  I can tell by the presence of the seven digit pairing code at the top of the screen:

Photo May 30, 3 43 35 PM.jpg

The seven digit “Pairing Code” allows me to connect this cab ride to my Arro account

Once I have the pairing code, I fire up the Arro app and punch in the number:

Photo Jun 01, 5 16 25 PM.png

Now I just add in the seven digit pair code to the app.

And that’s it… I have paired.  Arro automatically selects my stored credit card and default tip amount of 20%:

Photo May 30, 3 44 27 PM.png

I can maintain multiple credit cards if I need to use different cards for work versus personal.

When all is done, I get an email, just like Uber:

Monosnap 2016-06-01 17-23-06.png

So, if you are in NYC or Boston and frequently ride in taxis, give it a try. It really does work well.


Posted in Uncategorized | Tagged , , , , | Leave a comment

Am I really saving money as a cord cutter?

A little over five years ago, we cut the cable cord. I had analyzed our television watching, and we really just watched a handful of shows.  I concluded that if I set up a TiVo with an Over The Air (OTA) antenna, I could get most of those shows in high definition for free (except for the TiVo service fee).  The remaining shows I could purchase from iTunes and still come out ahead.

A lot has changed over those five years. The TiVo worked great for a while, but after a few years the quality of the OTA signal started to degrade. More and more channels had a problems coming in. In the meantime, the Apple TV came out, and the functionality of Hulu’s service improved.

About two years ago, we dropped the TiVo altogether and just went straight internet only. Most shows we could find on Hulu, except for some premium content like Game of Thrones and Doctor Who which had to be purchased on iTunes.

And then there was the problem of CBS. While all the other broadcast channels were part of Hulu, CBS decided to go it alone and not put their content on the streaming site. Unfortunately, several of my wife’s favorite programs like NCIS are on CBS, so we were forced to buy them on iTunes.  Still, there were just a few of them, and the season pass subscriptions still amounted to just about one month of cable.

Here is where things took an odd turn. Hulu was full of commercials, which was particularly annoying given that we were already paying to get the service in the first place. We had gotten spoiled by years of fast forwarding through commercials on the TiVo, and the iTunes shows had the benefit of being commercial free (which was good, considering that they were $2.99 for a high-definition show).  Each time a Hulu commercial would come on that we couldn’t fast forward through, we would grumble.

So, when Hulu rolled out a premium version of their plan that offered no commercials, I jumped at it. It was only $4 more a month ($11.99 up from the regular $7.99), and I never looked back.

A few months ago, CBS rolled out their own digital stream service, called CBS All Access.  It was $6.99 per month, but that was still much cheaper than buying the episodes on iTunes. I immediately signed us up for the free trial.

We tried it with great anticipation… and we were very disappointed. It was filled with commercials – constantly. It was essentially just like watching regular broadcast television with no TiVo capabilities. My wife gave me a look which could be loosely translated as, “Why do you get to watch all of your shows on Hulu with no commercials and I have to sit through all these commercials for mine?”

We canceled the CBS trial after two days and went back to iTunes.

This has been working pretty for us, and today out of curiosity I decided to look at just how much we were spending.  First, there were the streaming services:

  • Netflix (grandfathered pricing): $7.99 per month
  • Hulu No-Commercial Plan: $11.99 per month

Okay, so $20 a month for streaming services.  Then, there were all of the iTunes shows. These are a little hard to compute since we periodically buy a season pass which is a large hit all at once and then watched over the course of the year. However, on the assumption that it evens out, I decided to look at a three month period.

It’s not easy to track iTunes purchases, but it turns out there is a hidden way. If you go to, you can sign in and see all of your recent purchases. You can even sub-divide them by type (App, Music, TV Show, etc).

I pulled up my TV shows and looked at the last three months:

Report a Problem 2016-03-28 17-53-18.png

And the total? Over the last three months, we spent $182 on television programs. That’s about $60 a month.  Combined with the streaming services, we are running about $80 a month.

So, are we actually saving money? Once we ignore the promotional pricing that expires after a year, adding cable television service would cost somewhere between $80 and $130 a month, once you factor in taxes, fees, and renting the cable box.

So, worst case, we are breaking even.  The best case is we are saving money each month, but probably not more than $20-30.

However, there are some subtleties that put us a little further ahead.  Even if we still had cable, our iTunes charges wouldn’t drop to zero. One of us is traveling for work at least once a month, and we often download shows to our iPad to watch on the plane or train home. Even a streaming cable subscription wouldn’t help us here. We would still have to buy and download those shows.

Ultimately, cord cutting probably isn’t so much about saving money. We do save a little bit of money, but it’s really about control. We are only paying for what we want. Even if we only save a few dollars, they are dollars we feel good about.

Posted in Uncategorized | Tagged , , , , , , | Leave a comment

Answered: What is the project at the Artesani Playground on Soldiers Field Road?

Over the last few weeks, crews have shut down the parking lot at the Artesani Playground on Soldiers Field Road in Brighton, a neighborhood in Boston.  It is the most bizarre system of giant rubber tubes that seem to be filled with water:


My original theory was that they were preparing a new snow farm to collect and melt snow. After last year’s historic winter storm, the city used this parking lot as a place to dump snow, and then they spent months trying to melt it all.  I thought that perhaps the idea would be to pile snow on top of the tubes and then heat it at the bottom, but it seemed awfully late in the season to set it up, and we’ve had very little snow.

And then, when they inflated the tubes, it became clear that this project had nothing to do with snow, but I couldn’t figure out what it was. A google search didn’t turn up much either, and inquiries on social media mostly turned up other people wondering the same thing.

Since I always need to know how something works, I did some more research and figured I would share my results for other curious folks.

I finally found one obscure reference, tying it to a project to dredge the Charles River.  From a DCR press release:

Beginning on Tuesday, January 26, 2016 and continuing through Saturday, April 30, 2016, the Department of Conservation and Recreation (DCR) will be closing the parking lot between Artesani and Herter Parks on Soldiers Field Road in Boston to accommodate the Charles River Dredging Project. The parking lot will be off-limits to the public for parking during this time.

Okay, great. But what the heck were they doing?  What was the function of this elaborate contraption of rubber tubes and pipes?

Over the weekend, I went for a run along the Charles River, accompanied by my kids on their bike and scooter.  At the halfway point, we stopped at the Artesani Playground, and while my kids pretended to be monkeys playing on the jungle gym, I walked over to chat with a few workers who were checking out the tubes.

The men explained to me that this is an elaborate water filtering system.  Apparently there is a culvert up stream that dumped a large amount of silt into the river. The dredging project is trying to clear out this sediment, and to do it they have set up a pumping station that is sucking up the silt and sending it down a pipe in the river.  The pipe emerges here at the Artesani parking lot, where it enters this series of tubes.  The tubes separate out the water from the sediment, and then they pump the cleaned water back into the river.  When they are done in a few months, the tubes will be full of the sediment, the water will be preserved, and the river will be clear.

I have to say I have never seen anything quite like it. Sure enough, if you stand by the “intake” tube, you can hear rocks and other sediment flowing through the pipe, while the “outtake” tubes just sound like water flowing, so it seems to be doing its job.

I dug a little further, and found some more details about the source further up the river.  It is coming from a placed called the Faneuil Brook Outfall, and this project was put out for a bid for around $500,000.  I haven’t figured out exactly where the Faneuil Brook Outfall is, but I think I drove by a pumping station just a quarter mile or so upstream.

I did also find a posting from 2012 by a DCR intern named “Evan” describing a problem with a sandbar being created at the Faneuil Brook outfall:

Outside of the office, I investigated the subwatershed for pollution and sediment deposits that could end up in the Charles River.  A growing concern in the river is a sandbar that is forming and growing near an outfall from Faneuil Brook.  I went into the field yesterday in order to attempt to establish any major sources of pollution or sediment deposits.  Driving down the streets, I took pictures of sites that may be improved or be a source of pollution.  Fortunately, I did not find any major sources of pollution or what could be supplying additional sediment to the sandbar; unfortunately, I could not find a cause of the deposition for us to implement strategies in order to stop from continuing.

My daughter is worried that this is actually some kind of elaborate plot to pollute the river, since she sees the water flowing back in and is convinced it must be harmful chemicals. I keep trying to reassure her that I’m sure the DCR knows what it is doing and this is just clean water.

She seems unconvinced. But I feel much better, having solved the mystery.

Posted in Uncategorized | Tagged , , , | 1 Comment

Parents: create a treasured memory by recording your baby’s lullaby

A few months after I started dating the woman I would eventually marry, we made a visit to Boston over President’s Day weekend. One evening, we had dinner at the home of her long time family friends, which was a not so subtle vetting to make sure I was up to snuff. Fortunately, I was found worthy and found myself back again on President’s Day several years in a row.

In those first few years, we played a game of imaging a dream house, where money was no object, and everyone would design their own rooms. Back in our early twenties, we had wild ideas, including heated floors and oversized hot water heaters. A few years later, we were married, and President’s Day became a yearly tradition.

We just celebrated our 21st anniversary President’s Day dinner with these same friends, and I found myself remarking on how how far we had come. We were now joined by our two children, Ayelet age 8 and Rafael age 5. Also, we were describing a potential addition we are designing for our house, though one much more modest than any of the fantasies of our youth. Overall, many of the visions of our future as young newlyweds were coming to pass.

Ayelet was curious about the game I was describing of the fantasy house and wanted to try designing her own. We asked her about what would be in her bedroom, and what things might help her fall asleep.

She immediately responded, “A recording of Daddy singing my bedtime song, and a recording of Ima (hebrew for ‘mother’) singing the song her daddy sang to her at bedtime.”

My heart jumped into my throat. For years, I would sing the same Regina Spektor song to Ayelet, and Rafael had a special song I would sing by Vienna Teng. As the kids have gotten older, these songs have faded away and are hardly ever sung. She didn’t want recordings of the songs… she wanted recordings of us singing them.

There is nothing quite as special as holding a little baby in your arms and singing a favorite song as he or she falls asleep. Repeated night after night, it becomes a treasured memory of your time with your baby when a soothing voice and gentle bounce of the knees could make everything okay.

Thinking back now, I wish I had recorded myself singing those songs. It would be a wonderful gift to be able to give them when they are grown and have children of their own. I still remember the words and will go ahead and record them, but it’s not the same now that the kids are older.

So, parents whose children are still young enough to listen to the same lullaby, take advantage. Smart phones have an easy recording function… capture that memory and save it. Right now, it may feel silly, but in a few decades it will be a treasure.

Posted in Uncategorized | Tagged , , | 1 Comment

Speed up customer service by using Twitter instead of calling on the phone

Last weekend, I was booked on a Southwest flight from Boston to Nashville for a series of work meetings. Most of the Northeast had been crippled by a winter storm, but both Boston and Nashville just received glancing blows. My flight was still listed as on time.

However, when I checked-in to receive my boarding pass, I discovered that my Pre-Check status did not come through. Pre-Check is a wonderful service provided by the TSA that gives you an expedited and simplified security check when you arrive at the airport.

The way that it works is that you pay them $85 to do a background check and finger prints, and assuming everything looks good, they put you on a special approved list. When you arrive at the airport, you get to go through a dedicated Pre-Check line, and you don’t have to take off your shoes or belt.  You also don’t have to take out your laptop or remove the 3 oz liquids from your bag.  You just go through an old fashioned metal detector and you are off. It’s like flying used to be back in the 1990s.

While I don’t fly super often (most of my travel lately has been on the Acela between New York and Boston), I’ve come to depend on it for flying. Naturally, I was horrified at the idea that I might have to go through the slow line and take off my shoes – I know, it’s a first world problem.

After confirming that my account had the correct KTN # (the magic code that makes Pre-Check work), I picked up the phone to call Southwest. After jumping through a series of automated prompts, I was told that the wait time would be 116 minutes. I immediately remembered that thousands of flights were being canceled due to the storm, and I’m sure they were being flooded with people trying to rebook.

I wasn’t about to wait 116 minutes, so I decided to try twitter. I found their customer service account and tweeted at them:

My KTN # is listed on my profile, but Pre Check doesn’t show up on my boarding pass. How can I fix this?

Within two minutes, Southwest had responded asking me to “Direct Message” them my name, confirmation number, and KTN. We then proceeded to message back and forth troubleshooting the issue.

The representative’s initial thought was that perhaps my last name was causing a mismatch with the Pre-Check systems. My last name of Rothman-Shore is hyphenated. Normally, having such a distinctive last name is quite helpful, but for the airlines it is a headache. Despite the fact that it is 2016, most airline systems cannot handle a hyphen in a name (what are they running on the backend, IBM mainframes from the 1980s?), so usually I fly as either “Rothman Shore” or “Rothmanshore”.

Over Twitter, they changed my last name from “Rothman Shore” to “Rothmanshore”, but this didn’t seem to do the trick either. After trying many other ideas over the course of an hour, including comparing one of my wife’s past flights, since she was able to use Pre-Check on Southwest and has the same hyphenated last name, they ran out of ideas:

I’m sorry for all of the trouble. It might be worth following up with TSA to see how exactly your profile is formatted, and if they have any recommendations. -Daniel

I had assumed this would be a dead-end, but it turns out that the TSA has their own Twitter-based customer service account called @AskTSA. They responded almost immediately, but unfortunately they were going to take a little more time:

Thanks for the information provided, Jeremy. Unfortunately, the earliest we will be able to give you an answer is in one business day. We apologize for the inconvenience and we’ll get back to you as soon as we receive an update.

Sigh… at this point my flight was a few hours away, so I had to knuckle down and go through the regular security line. Hopefully we could sort out the issue and fix the problem for my return flight. The combination of a Sunday evening and the large number of canceled flights made for a rather empty airport, so aside from a little extra inconvenience, no harm done.

When we landed in Nashville Sunday evening, I turned on my phone to see a notification from TSA:

Hi Jeremy. Our records show the DOB in your reservation didn’t match the DOB in your TSA Precheck enrollment. Your DOB was submitted as [**redacted**] and our records list your DOB as [**redacted**]. If your full name, DOB, or KTN in your flight reservation isn’t an exact match to the information you enrolled with then you will not receive TSA Precheck status. Please reach out to your airline to have them update your DOB. When you reprint your boarding pass you should receive TSA Precheck status.

Well, that would do it! I waited at the end of the jetway for my other colleagues and tweeted back to Southwest to try to fix the issue.  Once again, they responded immediately, but now there was a new hurdle:

I see. Your birthday in your reservation was also listed as [**redacted**], but I assume this may have pre-populated from your Rewards account. In order for us to make the change within your account, can you send us a photo copy/picture of your ID for verification? -Rebeca

Umm… how was I supposed to to do that? You can’t send attachments on direct messages. And then, I remembered Dropbox. I snapped a photo of my driver’s license, uploaded it to Dropbox, and then tweeted it back to Southwest. Moments later:

All should be good now! Let us know how things go on your return trip. -Rebeca

Sure enough, a few days later, I checked in and my return flight to find that my Pre-Check status had indeed come through.

I’m still amazed by just how responsive the airline was over Twitter. On a day when wait times were pushing two hours, I was having real time messaging with their customer service department. Rather than being chained to the phone trying to sort it out, I was able to just message back and forth at my convenience, wherever I was. All I ever needed was my smart phone.

I think in the future, Twitter will be my first stop on customer service issues. Picking up the phone will be the last resort.

Posted in Uncategorized | Tagged , , , | Leave a comment

Putting a home repair theory to the test

About four and a half years ago, I posted about my belief that warranties usually aren’t worth the money. Our sump pump had failed, and the company that had installed it charged me $850 to put in a new one.  Had I paid $165 a year for the warranty and service visits, they would have replaced it for free, but by my math I still broke even.

At the time, I concluded that I could probably do this much cheaper myself, and I swore I would do so if this ever happened again:

More importantly, I watched everything the tech did when he replaced the sump pump.  It doesn’t look that complicated, now that I see how the pipes fit together.  If it fails again, I think I might take a stab installing a much cheaper one from Home Depot myself now that I know what to do.

Well, guess what? That sump pump just failed again. Time to put the theory to the test.

So many years later, I had a vague memory of how the old sump pump was connected from watching the tech replace it last time. It should just be a simple matter of loosening the collar that held the pipe on, and then I should be able to remove the pump and put in a new one.

First, I needed to actually get to the sump pump, which was covered with gardening supplies and other assorted basement items.  After clearing them all away and removing the cover, I peered in to the dimly lit tank to find the pump and see how well it matched up with my memories.

For tasks like these, I’ve found a very handy device… a neck mounted lamp called a HugLight.  It’s a pair of LED flashlights mounted on a semi-flexible cord that you can bend to hold in place, either hooked around an object or around your neck.  It let’s you keep both hands free but still have light right where you need it.

Web Image 2016-01-03 20-15-07.png

Once I got in there, it was just as I remembered.  I found the collar and with a screw driver I was able to disconnect and pull out the pump.  Easy.

Then, it was a trip to Home Depot.  I found the sump pumps in the back, at a much more reasonable range of $100 to $250.  I chose one for $230 because it looked the most similar to the one I had just removed.  Also, the last time I did this, it cost me $850, so even the most expensive models would still be a substantial savings.

Back at the house, I dropped the new sump pump in… and discovered that even though they looked similar, the pipe was a good 4-5 inches too short to reach the connector. Shoot, I hadn’t thought of that when I thought this would be so easy.

So, here I was at the turning point… I could either call in a plumber, or I could keep trying to fix this myself. I really like our plumber (Tom Lanza, if you need someone in the Cambridge/Somerville area).  The problem is that I would then need to juggle my schedule to arrange for him to come out.

Also, I don’t mind paying a plumber for complex jobs, but I hate spending the money on something that is actually quite easy, if you just know the steps. I still felt like this shouldn’t be too hard, so I persevered.

And so, back off to the Home Depot to see if I could find a connector.  Sure enough, there was exactly such a device for connecting a sump pump to an existing drain pipe.  It cost me another $25, but I was still way ahead.

Then came the challenging part – with the connector, the pipe was now too long, so I would need to use a hacksaw to cut the old pipe down to size.  I knew that if I messed this up, I would have to call in the plumber.

With a deep breath, I made the measurement and then cut the pipe. Within a few minutes, I had attached the connector and then hooked in the sump pump. A quick test showed that it was able to successfully pump the water out.

Success!  At least I hope so… it will have to wait until the next major rain storm to find out if I did this properly.

Posted in Uncategorized | Tagged , , , | 2 Comments

Star Wars makes great strides in creating strong female characters, but the toy makers lag far behind


Without revealing any spoilers about the new Star Wars movie, I can safely say this… they have made great strides in creating  strong female characters.

While there were initial concerns about there being relatively few female characters in the lineup, seeing the movie puts the fears to doubt. Daisy Ridley’s Rey is a great leap forward. She’s definitely a character that both girls and boys can get excited about.

Princess Leia was progressively strong for a female character as things stood back in the 1980’s, but she still needed a fair amount of rescuing. Rey, in contrast, breaks out into new territory and stands on her own as a heroic character.

As a father with a daughter, I’m excited to see Star Wars send this message. Not that I’m taking my daughter to see the movie… she gets too scared to even see the original movies.  However, she likes to hear about the story and it’s great to be able to tell her what a strong character Rey is.

Unfortunately, it seems that the toy makers haven’t taken up the charge in the same way.  One of the ways that the original Star Wars movie broke new ground was in their extensive merchandising tie-ins, particularly around the toys. Kids left the movie not wanting it to end, and the toys allowed them to keep playing the most exciting parts over and over.

This is still true in 2015, and the market is absolutely flooded with Star Wars themed toys. Kylo Ren’s face is plastered everywhere, and there are all sorts of figures and vehicles. However, I’m sad to say that Rey is not well represented.

My daughter was excited to hear about Rey’s strong character, and I thought it would be nice to get her a Rey figure to play with. I stopped off at a Toys R Us to see what was available. This close to Christmas, I expected the stock to be picked over, and sure enough it was.  Kylo Ren and BB-8 were nowhere to be found, but there were plenty of other characters around. There were lots of Finn (the male lead) and Poe Dameron (the rebel pilot). And there were all sorts of other vehicles and figures. But not a single Rey.

Now, it’s possible that all of the Rey figure were snapped up, leaving lots of Finns and Poes, but I’m doubtful. I followed the build up to the movie closely, and even I was surprised by what a strong character Rey turned out to be.  While it’s possible that Rey was manufactured in equal numbers to the other characters, I’m skeptical.  Finn and Poe were heavily cross-marketed in lots of items, but Rey was absent.  I’m pretty sure that the toy makers figured that Star Wars figures were for boys, and they wouldn’t need to manufacture as many Rey figures.

Thankfully, Amazon came to the rescue.  While I couldn’t find an individual Rey figure, I did find a set of six figures that included Rey.


Ayelet with her completed Rey coloring page

The good news is that there are more movies coming out in the next few years.  Hopefully Hasbro will get it right next time.

Posted in Uncategorized | Tagged , , , | Leave a comment