(If you don't know what I'm talking about, that's okay. This post is mostly for Google, in case someone else has a similar problem.)

The first controller was listed as a IBM ServeRAID M1015, which is a rebranded LSI 9240-8i. It offers two 4-lane SAS connectors, hence the -8i suffix. ZFS afficianados will recognize this as the controller of choice. It's cheap (~$90) because it was produced in vast quantities and is now being discarded in vast quantities. What's more, despite normally having onboard RAID functions, it can be cross-flashed into a non-RAID LSI 9211-8i HBA, giving the host direct access to all the drives – perfect for use with ZFS.

My M1015 arrived, and it was visually indistinguishable from a 9240-8i; there's literally no physical evidence of IBM anywhere. It's almost as if someone took an LSI card and flashed it to say "IBM M1015" in the MPT BIOS! Like, maybe… IBM.

It matters not. I followed the standard recipe to turn it into a generic LSI SAS2008 HBA, and life was good.

The second card was listed as an "HP LSI 9207-8e". This card is a straight-up SAS HBA with no RAID functions, which is what I want, and it has two four-lane external ports (-8e), which is what I want. Easy. It would work as-is, except I wanted to flash it to the same firmware level as the other LSI HBA, which is P16 to match the drivers in my target operating system.

On arrival, I noted that it had HP branding all over: the anti-static bag, the PCB silkscreen, everything. I removed the other card just to eliminate all possible confusion and booted up. The card's BIOS identifies it as an "HP 200 Series":

I dug in with sas2flsh -list, which says it's a SAS2308_2(B0) controller on a H221 board, which is definitely an HP part number.

First things first. I wanted to wipe the BIOS, since no matter what I won't be booting off this card. The SAS2Flash manual said boot support lives in memory region 5, so I wiped it with sas2flsh -o -e 5:

Next I told it to flash the 9207-8e P16 firmware, with sas2flsh -f 9207-8e.bin:

Huh. NVDATA Image does not match Controller Chip Revision, you say. No compatible NVDATA Image(s) found!, you say.

*go dig around on the internet*

Word on the street is that the HP H221 had a hardware revision, and while it's a rebranded LSI 9207-8e nowadays, some of the older ones floating around are in fact rebranded LSI 9205-8e cards.

So, next, I grabbed the 9205-8e P16 firmware and tried loading it with sas2flsh -f 9205-8e.bin:

NVDATA Device ID and Chip Revision match verified. Hooray!

ERROR: NVDATA Image does not match Controller SubSystem ID! Boo!

*go dig around on the internet some more*

The fact that the device ID and chip revision match means that I definitely have firmware for the right card. In other words, while it was readily apparent that this LSI 9207-8e was first sold as an HP H221, it turns out this HP H221 is really an LSI 9205-8e.

(Why did this change? Easy: the 9205-8e was discontinued and replaced by the 9207-8e. Both cards use the same LSI SAS2308 controller, but the 9207 supports PCI Express 3.0, while the 9205 does not. This limits the 9205 to ~2700 MB/s of throughput, which… seems okay for now.)

(Further aside: why would HP sell both under the same part number? Because they're jerks. If you change something in a way that breaks firmware compatibility without changing the part number, screw you.)

I don't know what the controller subsystem ID mismatch message means exactly, but somebody on the internet made it sound like that's an advisory check performed only by newer versions of the flashing utility. If that's the case, I figured such a mismatch couldn't be significant enough to brick the controller, so why not give it a try? I downloaded the P14 ZIP and pulled out the P14 sas2flsh.exe.

To reiterate: I decided to flash the P16 firmware with P14 utility. Sounds brilliant, right? What can possibly go wrong? Well…

It worked.

The vast majority of the people on the network are guests on wifi.

CSC can be broken down into three domains:

• The south building, including the Flight Deck restaurant, pro shop, bathrooms, showers, and team rooms
• The main hangar, which is largely open space but also contains several offices both ground-level and on a mezzanine
• Outdoors, including the CSCity RV park, the courtyard, the loading area, and outdoor dining at Flight Deck

Both the south building and the main hangar are served using three access points. Their locations, frequency assignments, and antenna coverage patterns were carefully chosen to avoid interference and to provide consistently high perforance, especially under load.

The couryard is easy enough to handle with omnis, but CSCity is a problem. Residents expect their cell phones to pick up the CSC wifi inside their trailers – even though trailers are basically Faraday cages. We can serve 5 GHz clients using directional antennas, but 2.4 GHz doesn't give us enough channels to make that a viable option, so we've been running with a single omni.

The last week included four separate projects:

1. We upgraded the CSCity mast on the SE corner of the main hangar. This mast provides service using two 14 dBi 5 GHz sector antennas and a 6 dBi 2.4 GHz omni. The antenna configuration did not change, but the radios powering it are improved.
2. We upgraded the courtyard mast on the NW corner of the south building. This mast provides service using a 10 dBi 5 GHz omni and a 6 dBi 2.4 GHz omni. As before, same antennas, better radios.
3. We added a CSCity North mast with a single 2.4 GHz access point. Channel selection was difficult; with the hangar door open, this AP can see at least one other AP on every frequency. I chose to alias the AP that serves the least traffic with the lowest transmit power, and CCQs remain high. CSCity North is the only AP that is not directly wired to the rest of the network using Ethernet; instead, it's connected via a point-to-multipoint 5 GHz backhaul.
4. We renumbered the public VLAN. We have been close to running out of addresses on a single /24; it's now a /22. IPv6 service remains unchanged at a /64, and DHCPv6 prefix delegation lets you request additional routed /63s.

I'm pretty happy with the performance during this event:

The main hangar access point has the highest load (see all those people on the packing mat?) but performance is excellent: 0.1% retransmit rate, 22 mbps throughput, 60 associated stations. Note also this access point is using only 11 dBm EIRP.

Other drop zones (*cough*) have networking poor enough that I bring a 3G hotspot. CSC's network makes me happy to live here full-time.

Follow the Velocity Flight Characteristics link. You'll see some wing loading recommendations:

Size does matter:

… [a Velocity] can be loaded as lightly as 1.0 pounds per square foot …

So what size is best for you?

… as long as you’re above a minimum of 1.0 pounds per square foot. …

All right, so the PD Velocity needs to be loaded above 1.0 lb/ft². I would have guessed higher.

Now, follow the Manual link. You'll get an undated Ram-Air Parachute Owner's Manual. Scroll to "Picking The Right Canopy" on page 14. This section includes the statement:

Performance Designs does not recommend that anyone jump a canopy that will result in the wing loading exceeding 1.0 lb./sq.ft.

All right, so PD does not recommend anything to be loaded above 1.0 lb/ft².

So… er… no one should jump a Velocity?

I made a video to celebrate the occasion:

Still nothing from Freefall Suits.

Note that I said "sells", not "makes".

TL;DR

I ordered and paid for a freefly suit and camera jacket from Chris Kotscha at Freefall Suits on September 21, 2013.

My five delivery dates:

• As of September 21: about 8 weeks
• As of December 12: "week of 13 January"
• As of February 7: "later next week"
• As of February 24: "The order WILL be completed and delivered before the season"
• As of March 19: "within the next 2 weeks"

I published this page after six months of getting nowhere.

Freefall Suits indicated that my freefly suit was completed on April 3, but I did not receive anything until April 19 – precisely 30 weeks after placing my order.

My camera jacket shipped May 2 but is still in transit.

The Whole Story

Quick links:

• Week 2
• Week 10
• Week 11
• Week 16
• Week 17
• Week 18
• Week 19
• Week 21
• Week 22
• Week 25
• Week 27
• Week 28
• Week 29
• Week 30
• Week 31
• Week 32

Initial contact

September 21, 2013

It was a Saturday. Like most Saturdays, I was skydiving, but with Nationals recently completed, I was fun-jumping instead of training with my team. It was my last weekend with my rental wingsuit, so I started the day with two wingsuit jumps before joining most of my friends doing other things.

I saw a tent go up after my third jump of the day. Ah, jumpsuits for sale! I was in the market for a freefly suit, so I made a point to stop in.

I popped over around jump #6 and said, literally, "sell me this jumpsuit". Chris – who I later learned was the founder of Free Fall Suits – took the demo suit by one arm and told me to pull on the other one. "No, no, really pull on it!" I did. The suit didn't care at all.

Honestly, very impressive demonstration. He told me all about the fabrics (this panel uses a Cordura/Kevlar blend!), the zippers, the stitching, everything. It looked and felt like an exceptionally well-made jumpsuit, but it was priced at the bottom end of the spectrum. I challenged him; Chris explained that he wanted to grow his company rather than to make a bunch of profit day one, and the best way to do that is to saturate the market with his suits, which meant selling them cheap.

This seemed like an unusual strategy, but Chris seemed like an unusual fellow. "All right, I'm sold", and he started measuring. I selected both a freefly suit and a camera jacket.

Chris had been planning to make a wingsuit jump, so I made sure to review: jump run is north, we're landing north, no more than 90º onto final, there's power lines on all the roads, etc. This left his associate to run the booth. We chatted about the company: they're just starting out, they've been on a tour, they were just leaving Nationals at Skydive Chicago and thought they'd stop by, etc. Seemed reasonable enough.

I watched Chris land under his reserve. Bummer – apparently he had a malfunction, chopped, opened into reserve line twists, and dropped his handles in order to sort it out. He spent the next while trying to gather his rig together again while I kept jumping.

After sunset, we looped back together again to finalize things: colors, options, payment. I paid in full using a Capital One VISA and got an Intuit payment receipt by email at 7:19 PM.

I was told to expect delivery in about 8 weeks, which fell just after the end of my season. I figured it didn't much matter if the schedule slipped because I wouldn't need it until spring.

Week 2

October 10, 2013

I received an email:

Good afternoon,

Please note that we are having a delay as some of the materials for the generation 4 suits did not pass the stress tests.

As we use materials that are custom made we are not able to simply go to a fabric store and start making the suits.

I apologize for the delay and will inform you as soon as we resolve this.

Thank you,

Chris Kotscha
Freefall Suits

All right. Again, I can deal with a delay.

Week 10

December 4, 2013

We're now two weeks past the original delivery date. I wrote an email, quoting the delay message above:

I haven't heard anything in quite a while. Is everything still on hold? More directly, could you provide an updated delivery window?

I included a copy of my receipt for reference.

Week 11

December 10, 2013

No reply. I wrote another email:

Hello?

I'm now concerned by the lack of communication. Additionally, I note that we're coming to the close of the 90-day chargeback window.

Please talk to me.

--Will

December 11, 2013

I received:

Good morning Will,

I am unsure why you have not received a response from the office. I will look into the status of your suit.

Thank you,

Chris

Sent from my iPhone

December 12, 2013

I sent:

When should I expect to hear back on this?

--Will

I received:

Good evening will,

I asked the office and they say your suit will be done on the week of 13 January.

I thank you for your patience. We had an issue with our material supplier but have found a new source and are catching up to the cue.

Thank you,

Chris

Sent from my iPhone

Optimistic, I sent:

Excellent! I appreciate the update.

What's the delivery time look like after that? I'm thinking I want to order another suit once I get my current order and can check it for fit and such.

Thanks,

--Will

And received:

Shipping normally takes 3-8 business days.

Cheers!

Sent from my iPhone

Week 16

January 13, 2014

This is the delivery date provided on December 12. I wrote:

Are we still on track for shipping this week?

Thanks,

--Will

Week 17

January 22, 2014

I sent:

I am beyond tired of emailing your company without reply. Customer service is a solved problem. Even a to-do list would suffice: check your email, add the issue to the list, reply "I'll get back to you", and work down the list. Any issues left open at the end of the day get a second reply saying "I haven't forgotten". It's not acceptable to ignore direct inquiries.

--Will

I received:

Good Morning Will,

I apologize that you have not had a good experience. I am heading back to the office to fix a number of issues that arose once I had left. Unfortunately I had thought that the office was responding to you and others but it turned out that they were not doing there job. As such you were not responded to and for that I really apologize. I have let go two people as a result of this lack of communication as you were not the only one (unfortunately), the shipping is one example as I was told it would be ready to ship. I passed that on to you and then it turned out to not be correct. I am heading back to the office this week and will ensure your suit gets built asap, even if I sit down and sew it myself.

Again I am heading back to the office to fix this issue and finding out why the inspection did not catch both the options and the measurements. Unfortunately it seems that as soon as I left things started to fall apart. I will fit it. I cancelled all my plans for the next few months so I can stay in the office.

My motivation for starting this company was to provide a well built suit at a price that was much more reasonable then what the market currently is. Money was not at all a motivator as I planned on not making anything for the first 4 years. I just want to make better products at a better rate.

I really do thank you for your patience, I understand your frustration as you this should not have been an issue. All I can do at this stage is makes sure that you get a good value and experience. We made mistakes but I will do anything to correct the issues and get you the suit.

Thank you,

Chris Kotscha

I sent:

I didn't even know there were issues with my order. The last contact I had was the December 12 email giving last week as the shipping target.

I appreciate your attention.

--Will Glynn
Sent from my iPhone

Week 18

January 31, 2014

I received what seems like a mass email since it's approximately a duplicate of the one I got on January 22. I guess lots of other people are having problems.

Good Morning,

I wanted to reach out to you so you know what has been happening with your suit order.

There have been some issues recently that I am currently addressing. Unfortunately I had thought that the office was responding to you and others but it turned out that they were not doing their job. As such you were not responded to and for that I really apologize. I did not realize the scope of the issue until I got back to the office. I have let go two people as a result of this lack of communication as you were not the only one (unfortunately). It seems that as soon as I left things started to fall apart. I WILL fix it. I cancelled all my plans for the next few months so I can stay in the office in order to resolve the delivery times and communication to ensure you and future orders get exactly what they expect.

In the September/October timeframe we switched over to the generation 4 models of suit and the transition of this design took longer than we had expected. These suits are amazing in build and strength and I know you will be happy. We fixed the modeling issues that arose and are now in the final stages of “catch up”. I apologize that this build up of orders affected the delivery time on your suit, know that we are working 2 shifts a day to catch up on this backlog and we are almost caught up.

Please send your email inquiries to the info@freefallsuits.com address and someone Will get back to you shortly, Unfortunately I personally receive over 500 emails a day and sometimes they get lost, the office will always assist you.

My motivation for starting this company was to provide a well built suit at a price that was much more reasonable then what the market currently is. Money was not at all a motivator as I planned on not making anything for the first 4 years. I just want to make better products at a better rate. I thank you for the opportunity to

I really do thank you for your patience, I understand your frustration as you this should not have been an issue. All I can do at this stage is makes sure that you get a good value and experience. We made mistakes but I will do anything to correct the issues and ensure you are happy.

Thank you,

Chris Kotscha

Week 19

February 5, 2014

I sent:

Can you provide me with an updated delivery date?

--Will

February 7, 2014

I sent:

I don’t really know why I expected a reply this time, but I’m disappointed.

Look: I need to know if you’re going to make that jumpsuit and camera jacket. I placed the order on September 21, 2013 — that's 20 weeks ago tomorrow. Safety Day is 4 weeks away, the start of my skydiving season is 7 weeks away, and the local iFLY tunnel should have its grand opening around then as well. If you’re not going to fill my order, I need to get someone else working on it now.

--Will

I received:

Good morning Will,

Your suit will be ready later next week.

I apologize for the delay. Since I got back everyone has been working double shifts in order to catch up the backlog.

Personally I apologize for not responding on Wednesday. I thought that I had but obviously confused that with responding to someone else's request.

Thank you,

Chris

Concerned that he might have forgotten I ordered two things:

Excellent, thank you.

Is this just the freefly suit, or does that include the camera jacket as well?

--Will

I received:

Hi Will,

That's for both. I thank you for your patience through these growing pains.

Sent from iPhone. Please note if I do not reply quickly I may have not received your email due to the high volume I receive.

Week 21

February 21, 2014

I haven’t heard anything in two weeks, so all I can do is assume there’s been no progress.

As a reminder:

Look: I need to know if you’re going to make that jumpsuit and camera jacket. I placed the order on September 21, 2013 — that's 20 weeks ago tomorrow. Safety Day is 4 weeks away, the start of my skydiving season is 7 weeks away, and the local iFLY tunnel should have its grand opening around then as well. If you’re not going to fill my order, I need to get someone else working on it now.

Tomorrow marks 22 weeks. It also marks one month since you offered:

I am heading back to the office this week and will ensure your suit gets built asap, even if I sit down and sew it myself.

—Will

Week 22

February 24, 2014

Hi Will,

I am trying to figure out an exact delivery date. The order WILL be completed and delivered before the season.

Unfortunately we are running behind on production. We recently experienced an issue with our made to measure software. This caused some grief as the issue resulted in ankles, wrists and the waist to be a bit smaller then the pattern. Because of this we had to redo a number of suits before they could be sent out. This compounded the other lessons learnt but we have learnt and continue to improve.

We have identified the issue and are working hard to catch up the cue as quickly as possible.

This delay in production caused the cue to build up. I apologize that this build up of orders affected the delivery time on your suit. I am still trying to get specific delivery dates to everyone, it's just going to take a few days for me to figure out how quickly we can catch up.

My motivation for starting this company was to provide a well built suit at a price that was more reasonable then what was on the market. We have experienced some growing pains but that motivation is as strong as ever.

I really do thank you for your patience.

Chris
Freefall Suits

Week 25

March 19, 2014

I wrote:

I am trying to figure out an exact delivery date. The order WILL be completed and delivered before the season.

Opening day is next Friday, March 28. I’ll have my suits by then?

—Will Glynn

Then I saw this on their Facebook page. They're both selling stock suits and inviting more custom orders while my 6-month-old paid-in-full custom order is still sitting unfulfilled. I had to comment:

Freefall Suits removed my comment but could not be bothered to reply to my email or otherwise address my situation. I found this extremely distasteful, so:

• I published this page to the web at large
• I shared this on dropzone.com, finding I'm not alone
• I shared this on Twitter and Google+
• I shared this with my local DZO and recommended against Freefall Suits as a vendor
• I sent this page to all my skydiving friends, some of whom have heard about Freefall Suits, but most of whom haven't

Freefall Suits: fill my order. Chris Kotscha: you said you would build my suit "asap, even if I sit down and sew it myself". That was two months ago. My season starts next Friday.

After publishing this, I received:

Good evening Will,

It will be close but you will have them within the next 2 weeks. We had to re-make your suit a few times as they didn't reach my standard.

I'm sorry it has taken so long, we ran into every administrative and personnel problem that I had ever thought possible. We are sorting it out but I know the wait was too long. We are working very very hard to get a fast turn around.

Thank you,

Chris

Sent from iPhone. Please note if I do not reply quickly I may have not received your email due to the high volume I receive.

I replied:

Thank you for the response. I look forward to receiving the suits.

You should know: I wrote you the email earlier today, and then saw a Facebook post inviting more custom orders. I posted a comment indicating my 6-month-old order is still in limbo – clearly as a frustrated customer – and instead of responding in any way, someone simply deleted it.

Deleting that one little comment made me feel not just ignored but actively silenced, so between then and now, I chronicled my entire Freefall Suits experience in full and published it to my website.

http://www.willglynn.com/2014/03/19/free-fall-suits-malice-or-incompetence/#week25

Delays are understandable, but combining both extreme delays and poor communication is a problem. I’ve had to reach out to Freefall Suits over and over again to get any kind of information, including today, and the responses I’ve gotten have mostly been broken promises.

I sincerely hope I will be able to end that post in a positive way, having received well-built, well-fitting jumpsuits at a fair price.

--Will

I received two more messages after I went to bed:

Hi Will,

I apologize that you got that impression. We have someone managing the Facebook that doesn't have any day to day information. When he saw your post he sent me a message telling me that no one reached out to you.

Unfortunately we are in our first year and have had personnel issues that caused communication issues. Obviously that still needs to be worked out.

All I want to do is provide a great suit at a way lower price then what the market currently is. I'm not making any money (actually loosing a ton right now) as that's not my motivation. Unfortunately it's hard to get others to understand the importance of answering the phone or messages instead of the physical cutting and sewing portions.

Hopefully this summer we will grow big enough to be able to afford front desk personnel to ensure effective communication.

Thank you,

Chris

Sent from iPhone. Please note if I do not reply quickly I may have not received your email due to the high volume I receive.

Followed by:

After reading your posts I realized that I had said I would sit down and sew it myself.

In my schedule I realized that it would be a wait if I did it myself as I suffer from a mountain of work. As such I looked at other options to speed things up. We built a pattern for your suit and gave it and the materials to a local seamstress. Expensive but speed was key. Since then the seamstress went on a very long vacation and didnt tell me until I received an email 2 weeks after I have everything over.

Not an excuse, our problems are ours to deal with. I just wanted you to know why I didn't physically sew the suit myself. After 4 months of working 7am until midnight or past I have realized that time for me to sew is virtually non existent.

One of the unfortunate realities of any new business is balancing employees, production, suppliers and infrastructure and bringing it all together. The easy way out would be to stop making suits, but I want this to succeed. It frustrates me to no one when people I know spend $1000 for a jumpsuit and I want to change that.

I thank you for your time and patience.

Chris

Sent from iPhone. Please note if I do not reply quickly I may have not received your email due to the high volume I receive.

March 20, 2014

I replied:

We have someone managing the Facebook that doesn't have any day to day information.

I found the Facebook post aggravating for two reasons: first, it happily invites readers to order more custom suits, even though I know with certainty that Freefall Suits has custom orders from at least six months ago still outstanding. Second, my comment was deleted very quickly – much more quickly than any other contact I’ve had with Freefall Suits – and it was deleted without any other form of contact, such as a private message indicating that someone would get back to me soon.

Unfortunately it's hard to get others to understand the importance of answering the phone or messages instead of the physical cutting and sewing portions.

It’s really about expectations. Perhaps showing these people my perspective as a customer will help them understand.

Hopefully this summer we will grow big enough to be able to afford front desk personnel to ensure effective communication.

I made some suggestions in my January 22 email where I assert that customer service is a solved problem. I believe you need a *system* — really, just about any system — not more people.

If you had told me in September that my suit wouldn’t be ready until March, that would have been completely fine. Unfortunately, I’ve let Freefall Suits set my expectations at each point, only to have each delivery window come and go without another word. I think someone should keep track of the delivery dates given to customers. You know a week in advance when a suit won't be ready in time, so you can be proactive — give your customer an update before you’re late.

Without that, we get my experience: the dates come and go, so I check to see what’s going on because no one has told me. If I got a reply setting expectations again, I’d be a little irritated, but generally okay. However, I didn't get anything from Freefall Suits unless I hassled you repeatedly; see my emails on December 4/10/12, January 13/22, February 5/7. Seriously? It’s not hard to figure out which emails need action and mark them as such. I’d even settle for “I’ll get back to you shortly”, but being totally silent after blowing a self-imposed deadline is just awful.

Whatever you’re doing now isn’t working. Fix it. There are plenty of tools available.

After reading your posts I realized that I had said I would sit down and sew it myself.

It really doesn’t matter to me if you personally construct my suit. Ultimately, you took my measurements and you run the company; if I have a problem with the results, I’ll be contacting you to resolve it regardless of who actually put it together.

That said, I did interpret this as meaning you were personally taking ownership of my order, and I took it at face value when you said you would “ensure [my] suit gets built asap”. That was two months ago. I suspect it would have been possible to start and finish producing my order in less than two months.

For comparison, a friend and I each ordered a wingsuit from Phoenix-Fly in November — exactly two months after I ordered from Free Fall Suits. We both received our orders on time in early February, right around when you told me “later next week”. My wingsuit was perfect, but his wingsuit had enough problems that Phoenix agreed to re-make it from scratch. Correcting this required a single phone call, and his second suit arrived three weeks ago.

For an additional comparison, I ordered a container from Rigging Innovations in October. I’m sure I was told when to expect delivery when I sent in the form but I didn’t remember anything more specific than “Q1 2014”. I wrote an email on January 9 at 4:53 PM:

I haven't heard anything in a while and I don't really remember when to expect production to be complete. Could you check on my order?

I got a response at 5:13 PM:

It's moving along, I'm guessing it should be done within 3-4 more weeks. The factory is closed for 2 weeks at the end of December, so that always creates a bit of a backlog for us. I attached a picture of it. Thanks!

I was so pleased, I sent the photo of my partially-constructed container to a bunch of people right then. It shipped three and a half weeks later. I know who I’ll talk to next time.

One of the unfortunate realities of any new business is balancing employees, production, suppliers and infrastructure and bringing it all together. The easy way out would be to stop making suits, but I want this to succeed. It frustrates me to no one when people I know spend $1000 for a jumpsuit and I want to change that.

I want you to succeed. I want to like your product enough to buy more and to recommend it to my friends. So: please, make me a happy customer. Fill my order and let’s go from there.

—Will

Week 27

March 29, 2014

My skydiving season officially started. This means we're also officially past my fourth delivery date, given on February 24:

The order WILL be completed and delivered before the season.

April 2, 2014

We're now at my fifth and most recent delivery date. I sent:

On Mar 19, 2014, at 10:00 PM, Chris Freefall suits wrote:

It will be close but you will have them within the next 2 weeks.

It's been exactly two weeks and I don't have the suits. I also don't have a tracking number, which makes me think they haven't been shipped, and I haven't received any information about my order's production, which makes me think nothing has happened at all.

Eight weeks ago:

Look: I need to know if you’re going to make that jumpsuit and camera jacket. I placed the order on September 21, 2013 — that's 20 weeks ago tomorrow. Safety Day is 4 weeks away, the start of my skydiving season is 7 weeks away, and the local iFLY tunnel should have its grand opening around then as well. If you’re not going to fill my order, I need to get someone else working on it now.

Not only did you agree to make these items, you said they would be ready "later next week".

I need these suits. You've repeatedly committed to providing them. Where are they?

--Will

I checked in on the Dropzone.com thread, finding posts from several other people still waiting for their orders as well as a post by a Freefall Suits representative. I replied:

We are currently processing the orders from the very end of Dec. Within the next 5 weeks we will be caught up on all the orders from January, February and March bringing us back to a 6.5 week delivery time.

If that's true then you're not processing orders chronologically. I ordered on September 21, and as of my fifth delivery date (today), I haven't received anything.

For perspective other suit companies have had a 24 week delivery time before when they were newer and learning lessons

Maybe so, but I'd guess those other companies don't make a habit of promising and failing to deliver. Again, I'm at five delivery dates blown -- delivery dates set by Freefall Suits. If you had quoted me a 24-week lead time, I probably would have been okay with that, but instead I was told I'd have my order in 8 weeks, 4 weeks, next week, 4 weeks, and 2 weeks.

As of now, my order is 27 weeks old and I don't even have an ETA.

April 3, 2014

I received this in the wee hours of the morning:

Hi Will,

Your freefly suit is done, now we need to work on your camera jacket.

I know we have broken past deadlines that we supplied. At the time for all of those I thought it was achievable, then we would run into more issues that delayed delivery.

Unfortunately due to a flurry of posts on dz.com we went from the usual 30-40hrs a week of answering phone / emails to an absurd amount. This meant that I had to pull a seamstress to answer phones in addition to myself and one other sharing the workload. As we are a small company this resulted in a loss of 15% of our production until the inquiries die down. When I quoted the 2 weeks I didn't predict that I would lose 60hrs of productive sewing labor in a week. Unfortunately hiring more staff is not a possibility to cover this off.

I used to think that building suits was easy. Frankly the physical sewing is easy, but getting the man power, infrastructure, supplies and administration together while concurrently trouble shooting each day is a challenge.

We have / are sending over 140 suits this week so we have been hyper productive to catch up after an endless list of issues (almost all externally influenced).

I thank you for your patience. We're a startup and people's patience has a benefit that I cannot begin to describe my appreciation of.

Chris

I replied:

Thank you for the update. I look forward to receiving the suits. Please send me a tracking number when they ship.

--Will

Week 28

April 10, 2014

A brand new account posted on dropzone.com that they ordered from Freefly Suits in late October, received a suit in February, sent it back for fixes, and received the altered suit in two weeks.

April 11, 2014

I replied on dropzone.com:

I jump in eloy and Chris was out there at the Halloween Boogie. … My suit came in February. … He took the suit that day and reenginered it . 2 Weeks later I had my suit back.

I ordered in September -- 29 weeks ago tomorrow -- and still have received nothing. According to Chris, my freefly suit was completed at least 8 days ago, but I haven't received a shipment or a shipment notification. Maybe he's waiting to finish my camera jacket too in order to save on postage...

But Freefall Suit makes unbelievable skydiving suits.

I hope so.

This story is consistent with my earlier suspicion that Freefall Suits doesn't have anything resembling a first-in, first-out production queue.

Week 29

April 12, 2014

I sent an email:

On Apr 3, 2014, at 12:39 AM, Chris Freefall suits wrote:

Your freefly suit is done, now we need to work on your camera jacket.

It's been 9 days since your last message, so I assume the camera jacket isn't ready yet. However, I need a freefly suit more than I need a camera jacket.

I understand that it would be cheaper to ship both together, but seeing as how my order is now 29 weeks old and my skydiving season is underway, I think it would be reasonable for you to ship both items as they become available. Could you ship my freefly suit on Monday?

--Will

I posted on dropzone.com:

What happens when you need more money, but there are no new orders?

Apparently you do a stock sale, solicit more new orders, and delete any negative feedback.

Chris indicated that my freefly suit was done a week and a half ago. I asked him to ship it on Monday even if the camera jacket I ordered is still outstanding. Let's see what happens next.

I received a reply:

Good afternoon Will,

I will ensure it is sent out on Monday.

Thank you,

Chris

April 14, 2014

Monday's business hours came and went without a word. Shortly before going to bed, I sent:

I will ensure it is sent out on Monday.

Was my freefly suit shipped today? If so, could you provide the tracking number?

--Will Glynn
Sent from my iPhone

A couple hours later (after midnight my time), I received:

Good evening Will,

The tracking number is USPS 9114 9012 3080 3140 4494 32

Thank you,

Chris

April 15, 2014

The tracking number shows it was supposed to be delivered today but the tracking data casts doubt on this prediction:

April 16, 2014

USPS continues to list an Expected Delivery Day of yesterday, but at least the package appears to be making progress:

This is consistent with USPS FY2014 Q1 statistics showing 75.3% on-time performance to Chicago and 84.0% overall.

Chris posted to the dropzone.com thread, addressing numerous issues. I responded:

Will - your suit was shipped and the tracking number was sent to your email.

That's true; I have a tracking number. I've been keeping my post up-to-date. If I receive the freefly suit, and it fits, and it's the correct colors, and it has the correct options, it will have been a 29-week ordeal -- and if there's a problem, it will end up being an even longer ordeal.

Either way, my Freefall Suits camera jacket is still in week 29 with no ETA.

April 17, 2014

No movement from USPS. "Expected Delivery Day" remains April 15.

The USPS performance report says "98.5 percent delivered within the service standard plus three days", which should be tomorrow. On the other hand, USPS appears to lose more packages than its competitors. Let's hope I get lucky.

April 18, 2014

No movement from USPS. I'm slated to organize at today's RW event, so I won't be sitting at my mailbox in Chicago all day, but I've arranged for a friend to stop by and collect the package should USPS indicate it was delivered.

Week 30

April 19, 2014

The USPS package was actually delivered to my home in Chicago today. I'm at the drop zone all weekend (team training!), so I had a friend stop by, retrieve the package from the entryway, place it inside my home, and send me a photo.

I plan to open the box on Monday.

April 21, 2014

Closer examination of the box reveals why it took so long: the handwritten shipping label contained the wrong ZIP code.

I opened the box, revealing a freefly suit in my colors and approximating my size. It has magnets, which is the option I cared most about. There are some issues with the fit, some of which I can't precisely diagnose alone.

My plan is to get feedback from my local rigger and prepare a full report tomorrow.

April 23, 2014

I visited my local rigging loft and we decided that the best next step is to jump the suit and see what happens. The weather should permit that on Friday.

April 25, 2014

I jumped the freefly suit a couple times. Positives:

• I fit inside of the suit.
• It's the right colors.
• It's warm.
• The arm material seems like it'll withstand contact with velco, say, from an altimeter wrist strap. (This is a source of wear on my current RW suit.)

Negatives:

• The torso and legs seem too long in general, both by about an inch and a half. The legs sort of bunch up above my shoes and mostly the excess goes away once I put on a rig. The torso, on the other hand, has a habit of riding up while I'm under canopy and making the zipper brush against my chin.
• The seams are bulky and don't really lay flat, instead developing an uneven waviness. (I think that's part of the design and not anything particular to my suit.) I'm calling this a negative because there are bulky, not-really-laying-flat vertical seams cuddled up right next to my pillow cutaway/reserve handles.
• The elastic around the ankles is small enough that it's legitimately a struggle to fit my feet through the leg holes. Maybe I have an unusual foot-to-ankle ratio?
• The collar is loose enough to breathe more than desired in freefall, though thankfully not so loose that it flaps around enough to irritate my skin.
• The magnets on the zipper are mounted in thin fabric and they readily snap to each other when the suit is not zipped.
• It'll probably be too warm in the summer. I needed gloves in the air today, and despite not wearing layers under the suit, I was almost sweating.

Despite the length of list, on the balance, I'm not dissatisfied with the suit for the price I paid. However, given the level of service I received during production, I am not inclined to ask Freefall Suits for alterations on this suit, or for additional jumpsuits in the future.

Now all I need is a camera jacket and I can be done with this company.

Week 31

April 28, 2014

Someone resurrected the Dropzone.com thread, calling out Freefall Suits' earlier "within the next 5 weeks we will be caught up" statement, saying:

Well, 5-1/2 more weeks have passed and still no suit. This takes my wait to just over 30 weeks so far.

I posted, citing my earlier reaction to this claim, adding:

To their credit, Freefall Suits filled half my order – I did, in fact, receive a freefly suit. That said, my camera jacket (ordered that same day in September, now 31 weeks ago) is still outstanding. Freefall Suits seems to have stopped providing me ETAs; I've heard nothing about my camera jacket since an April 3 email which seemed to imply that production hadn't started.

April 29, 2014

Four business hours after my forum post, I received:

Good morning Will,

I am following up as we haven't received a reply in a week.

Your camera jacket is finished and I will send a tracking number as soon as I get it from S&R.

I thank you for your patience as we worked through our issues.

Chris

I replied:

Good morning Will,

I am following up as we haven't received a reply in a week.

It took a week to receive the shipment. I wasn't able to jump the freefly suit until Friday.

Your camera jacket is finished and I will send a tracking number as soon as I get it from S&R.

I thank you for your patience as we worked through our issues.

Excellent, thank you.

--Will

May 2, 2014

Another customer posted on Dropzone.com, complaining about splipping delivery dates and poor communication, including his correspondence as well. I replied:

I am just going to start posting my correspondence with this company in this thread, since your blog seems to have gotten you results maybe this will help.

Posting here seems to get attention too.

I had no contact with Freefall Suits since an April 14 email exchange. I posted here two weeks later, and within hours Chris emailed me saying my camera jacket is complete, promising a tracking number. Now, it's been three days and I don't have a tracking number, but still – this was the first ever unprompted email I've received from Freefall Suits, and it happened right after post #37.

Customers shouldn't have to resort to public shaming.

I then received an email:

Good morning Will,

Tracking number USPS 9114 9012 3080 3140 3771 79

Thank you,

Chris

I replied:

Thank you. I look forward to its arrival.

--Will

I also posted to Dropzone.com:

Now, it's been three days and I don't have a tracking number, but still – this was the first ever unprompted email I've received from Freefall Suits, and it happened right after post #37.

…and now, a few hours after posting this, I received a tracking number.

Customers shouldn't have to resort to public shaming.

Seems like it works.

I received another email:

I'm genuinely sorry it took as long as it did. Never in my life has I faced so many hurdles to getting things done. After tons of hard work from everyone, double shifts for months on end we are finally catching up on the cue. A few other suits are late but we are rapidly catching up on clearing out the entire cue.

Thank you for your patience as we sorted out the growing pains.

I replied:

On May 2, 2014, at 1:31 PM, Chris Kotscha wrote:

I'm genuinely sorry it took as long as it did. Never in my life has I faced so many hurdles to getting things done. After tons of hard work from everyone, double shifts for months on end we are finally catching up on the cue. A few other suits are late but we are rapidly catching up on clearing out the entire cue.

Thank you for your patience as we sorted out the growing pains.

That's great. I hope you can deliver your product at your price with an acceptable level of service – you could actually force the jumpsuit industry to compete. Your success would benefit everyone.

However: I have stopped caring. I want to be done.

Every single opportunity Freefall Suits has had to impress me has resulted in disappointment.

I placed an order; you could impress me by filling it promptly. Then there was a problem with materials and production was halted: all right, so impress me with how you handle that by keeping me in the loop with how things are unfolding. Then I wrote an email concerned by the lack of communication: you could impress me by telling me what's going on and instilling confidence in your ability to handle things going forward. Then I wrote another email annoyed that no one replied: you could impress me by taking control of the customer service situation. On and on and on, weeks and weeks and weeks.

You did impress me at one point after making the sale:

I am heading back to the office this week and will ensure your suit gets built asap, even if I sit down and sew it myself.

…and then disappointed me further as my order continued to sit unfilled.

At one point, I told you what I need as directly as possible:

Look: I need to know if you’re going to make that jumpsuit and camera jacket. I placed the order on September 21, 2013 — that's 20 weeks ago tomorrow. Safety Day is 4 weeks away, the start of my skydiving season is 7 weeks away, and the local iFLY tunnel should have its grand opening around then as well. If you’re not going to fill my order, I need to get someone else working on it now.

You replied:

Your suit will be ready later next week.

That was 12 weeks ago today.

The reasons don't matter, the apologies don't help, and my understanding doesn't change anything. Ultimately, "disappointed" doesn't begin to capture my experience.

I went public when the frustration of dealing with your company was too much to bear. It took a lot to get me to that place. I've been disappointed by plenty of companies, even in skydiving: just three weeks ago, a canopy I ordered over the winter missed its delivery date, at which point we all found out that the factory hadn't even started production. They offered me a reason to wait a second time and gave me the information I needed to make a decision. I ended up ordering a competing product instead, but I'm not even upset. I'll happily talk to them again next time I need a canopy.

Freefall Suits, on the other hand, has taken "ordeal" to a whole new level. I believe I've accurately chronicled my entire customer experience, I believe it speaks for itself, and I believe people ought to know that this happened September 2013 through May 2014.

I want to receive that camera jacket. I want it to fit, I want it to be in the correct colors, and I want it to have the correct options. That's it. I want to be done.

--Will

Week 32

May 3, 2014

USPS inexplicably reports that the package is "available for pickup". I hope they bother trying to deliver it at some point because a) it's their job and b) my local post office is a frustrating place to visit.

It's fitting that Freefall Suits would ship with USPS.

May 5, 2014

I filled out the USPS problem form around 11 AM:

Topic: Receiving Mail > No Delivery/No Attempt > I Did Not Receive a Specific Item

Tracking number: 9114901230803140377179

Message: This package arrived at the 60640 post office and is presently "Available for Pickup", but no delivery has been attempted. I would like USPS to deliver this package.

I selected that I want a response by email instead of a response by phone:

However, my local post office has disregarded this selection in every previous instance, choosing to call me or to ignore the issue instead.

My case number is HQ117780154.

I received an aggravating phone call just before 6 PM from a woman at the Uptown post office. She did not bother reading any of my report and instead asked me to provide all my information again. She indicated she would "look for the package in the place where we keep packages" (essentially what I asked them to do in the first place) and call me back tomorrow.

Around 8 PM, the tracking updated to show delivery earlier this afternoon. I'm not in Chicago, but my wife is, and she retrieved the box from my building's lobby. This package has the correct ZIP code.

I uploaded a sample project as a demonstration. Here's the .travis.yml:

---
language: c
before_install:
  - sudo apt-get update -qq
install:
  - sudo apt-get install -qq gcc-avr binutils-avr avr-libc
script: make

Open firmware is one of the reasons I bought my FlySight audible GPS. Open firmware means I can hack on it, sure, but it also lets me use world-class tools like GitHub and Travis-CI.

Of course, the Wayback Machine has copies of nearly everything, and this site is no exception. The content isn't gone to the sands of time – but it's also nowhere near accessible, since the individual crawls presented by the Wayback Machine are various fragments.

I want to get the data out of the Wayback Machine, process it into a cohesive form, and give it back to the creator so he can make it available on the web once again.

Getting the data out poses two problems. First, I want to know exactly what data the Wayback Machine contains. Second, I want to retrieve the items of interest.

Querying

The Wayback Machine is basically a big distributed filesystem that contains 5+ petabytes of archived content. The data is stored in WARC files, each weighing about a gigabyte.

WARCs (and ARCs before them) are just blobs containing record after record after record with no features to support random access. That's a problem, so the Wayback Machine generates secondary indexes called CDX files. These allow a reader to jump straight to a particular record without having to read (and decompress!) an entire gigabyte of crawl results.

The Internet Archive offers a CDX Server API. This allows you to phrase a query and get back a list of records that match. For example, here's the first record for google.com:

com,google)/ 19981111184551 http://google.com:80/ text/html 200 HOQ2TGPYAEQJPNUA6M4SMZ3NGQRBXDZ3 381

This tells us more than enough: the URL, timestamp, HTTP status code, path, content type, hash, and length.

In short, the CDX server allows me to get a list of archived resources with enough detail for me to decide if it's an object I need.

Retrieval

This is where it gets harder. See, the Wayback Machine is designed for use by humans, but I want to hook it into a machine.

Check out the Wayback Machine for google.com, 1998-11-11:

See the bar up top? View source:

Ewww. The Wayback Machine added 250 lines of not-quite-HTML. This makes it easy for humans to navigate, but it complicates machine parsing.

I dug around the Wayback machine source code and discovered a solution. The /web/YYYYMMDDHHMMSS/<url> syntax accepts flags after the timestamp. This is used to trigger certain filters from various contexts: <link rel="stylesheet"> links are rewritten to trigger CSS-specific processing, <img> links are rewritten to trigger image-specific processing, <script> does Javascript, etc.

I discovered that this mechanism supports an id flag. This flag ultimately causes the resource to be handled by TransparentReplayRenderer, which attempts to hand back the content in an unmodified form.

View source on /web/19981111184551id_/http://www.google.com/:

Bingo!

I'd prefer a way to get the corresponding WARC record for maximum fidelity, but that's not available right now. The headers aren't quite preserved – some new ones get added and I think some of the originals get clobbered in various situations, but it's better than the alternatives. At least the HTTP response body isn't completely rewritten.

(There's another way to trigger this behavior too, but I'm not sure it's intentional and it might have security implications.)

Now, the question is: does archive.org intend for me to connect these two services? I'm guessing no – the terms of service grant access "for scholarship and research purposes only", and their robots.txt forbids machine access. Still, the existence of the CDX server indicates they support certain types of machine interaction, so I wrote them an email to ask. I'll keep you posted.

The problem

I have a project that involves a bunch of nginx servers in a bunch of different locations. If one of them stops working, I'd like to stop using it until it gets better. I'll be using Amazon Route 53 health checks to provide this behavior, but plenty of other things work the same way.

All I need to do is to make each server indicate if it's healthy or not. But, hmm… what does "healthy" mean in this context?

This project uses nginx as caching proxies. Most of the content is immediately available most of the time, but some of the time, nginx has to consult other systems over the network. So, even if the local system is working fine, it might not actually be capable of serving clients, in which case I want the health check to fail.

The status check is an HTTP GET request. The upstream system will interpret any 200 OK response as success, any any other response as a failure. That sounds like the logic I want.

Except… no, I need to check several paths. Crap.

Well, let's see. nginx is built around an asynchronous event processing loop, which internally supports the notion of sub-requests. All the health check needs to do is to make some sub-requests, wait for them to complete, and return 200 OK if and only if they all returned 200 OK. That totally fits the nginx request processing model, so this shouldn't be hard, right?

The solution

Well, turns out it wasn't hard. nginx supports running arbitrary logic in-process via the ngx_lua module, and this module exposes enough of the internals to do exactly what I want.

Specifically, I can use ngx.location.capture_multi() to run multiple simultaneous sub-requests, ngx.status = 500 to signal failure, and ngx.print() to return a request body just for good measure. It's been a long time since I wrote Lua, but:

location /health_check {
  default_type 'text/plain';
  content_by_lua '
    local reqs = {
      { "/" },
      { "/path1" },
      { "/path2" }
    }
    local resps = { ngx.location.capture_multi(reqs) }
    local ok = true

    for i, resp in ipairs(resps) do
      local req = reqs[i]
      table.insert(body, "- GET ")
      table.insert(body, req[1])
      table.insert(body, " returned ")
      table.insert(body, resp.status)
      table.insert(body, "\\n")

      if resp.status ~= 200 then
        ok = false
      end
    end

    table.insert(body, "\\n")

    if ok then
      table.insert(body, "ok\\n")
    else
      table.insert(body, "failed\\n")
      ngx.status = 500
    end

    ngx.print(table.concat(body))
  ';
}

This satisfies my goal of testing multiple paths (in parallel!) and returning 200 OK if and only if each of them are working.

Health check: accomplished.

As a bonus, /health_check returns a useful plain-text result should I care to inspect it from a browser:

- GET / returned 200
- GET /path1 returned 200
- GET /path2 returned 500

failed

Implementation notes

Note that I'm buffering the response in body rather than immediately calling ngx.print(). This is necessary because ngx.print() tells nginx to write the passed string as part of the response body, which implicitly tells it to send the response headers, including the status code. Therefore, I can't call ngx.print() before I've determined the status code I want to send.

Additionally, for those try-it-at-home types, be aware that ngx_lua is not an official part of the nginx source tree. It is, however, available in the nginx-extras package on Debian-related distributions, and is therefore also part of Ubuntu LTS, which is good enough for me.

I had a great year skydiving. It's sad that it's over, but I made the most of it. Special thanks to Chicagoland Skydiving Center for being awesome.

Airspeed

Returning to a Katana after the demo Crossfire 2, I find that I adore the increased airspeed. I can fly faster on this Katana 150 than I could on the Crossfire 2 149, due in large part to the lightness of the front risers.

That extra airpseed was extremely useful today. The winds up top were exceptionally strong: 27 knots ground speed on jump run, so something like 70 mph at 13000' and forecast to be something like 30 mph at 3000'. I was first out on first load, and I chose an exit point a little over a mile away. Freefall drift was such that I opened up slightly short but quite a ways crosswind, and the canopy winds were moving me backwards at full flight. I got on my fronts and managed to crab back to the airport with enough altitude to fly a landing pattern, but on the Crossfire 2 (or my Sabre 2), I would have been forced down in the corn.

Well, that's not true. If I had a slower canopy, I would have stayed on the ground. Knowing I had a Katana, I chose to jump on load one, trusting that the canopy had the ability to make progress even if I had to fly into the wind. Sure enough, I needed to do exactly that, and the Katana delivered.

Flight

I put four jumps on this canopy today.

It opens way better than the Katana 170 I demoed a month ago. The openings were my primary complaint with that Katana, but this one feels very different. Like the 170, this Katana 150 still searches when it snivels, but it inflates much more symmetrically.

It handles much like the Katana 170, though a bit snappier, which is expected due to the smaller size. Rears feel the same, fronts feel the same (and I love it!), though brakes feel a little different.

Brakes

I noticed that the brake lines look a little different too:

The Katana 170 didn't look like that, so this caught my attention. According to the Katana line trim chart, the upper steering lines should be arranged from longest to shortest, with the shortest lines towards the outside. I examined the canopy after the next jump, finding:

A-ha, proof! The innermost upper steering lines were switched with their neighbors when this canopy was re-lined. The trim chart indicates that this arrangement is not correct for any size Katana, so I'm really not sure what the rigger was doing.

Besides looking strange, I suspect this might be why the toggles don't respond quite the way I remember, and why the stall point isn't where I expected to find it.

My friendly local rigger (who did not perform the re-line) intends to take a look tomorrow. It should be a quick fix thanks to the bar tack machine, though Katanas use HMA lines, and HMA doesn't really like having stitches pulled out.

Toggle fire

So, again, I've made four jumps on this Katana 150 so far and three of the openings were far better than the Katana 170. The odd one out was a toggle fire. The parachute opened normally, but quickly started turning and diving just like I was pulling on one of the toggles – or, say, because one of the toggles came loose while the other was still stowed.

You can fix this by a) grabbing the loose toggle and pulling it or b) by un-stowing the still-stowed toggle. And sure enough, in my case, I easily arrested the spin once I got the toggles in my hands.

A toggle fire isn't terribly surprising. It's not normal for me, but these aren't my normal risers and these toggles don't stow the same way, so I must have done it wrong.

At least, that's what I thought.

See, the canopy felt fully inflated, then it started turning rather aggressively, exactly as if I was holding the right toggle to about hip level. That perfectly fits the profile of a toggle fire, so I went for the toggles. I focused on securing the left toggle, because that's the one that must have popped loose, but I ended up grabbing and releasing both at the same time.

Except… the left toggle isn't loose. It's firmly seated. You can see the steering line tucked onto the back of the riser, indicating it's still properly set. I can't say for sure what actually happened; it felt exactly like a toggle turn, and not a front riser, rear riser, or harness turn. Still, I can say for sure what didn't happen, and according to the video, the left toggle did not release before I pulled it.

This was not a toggle fire.

My best guess is that this was a tension knot involving the right brake line. The video neither supports nor refutes this hypothesis. Still, I was chasing a CRW group, so this was a slow (subterminal) deployment – prime territory for tension knots – and releasing both brakes as I did could conceivably clear a tension knot on the right side. It fits the evidence, but I just don't know.

Conclusion

I'm still on the fence about what to put in my new container come spring, but I am glad to be jumping this canopy. Also, it's for sale, and it does match my colors…

I pop open Activity Monitor, turn to the Disk Activity tab, and open Terminal:

$ cd /Volumes/Drobo
$ dd if=input_file of=/dev/null bs=$(( 1024 * 1024 ))

Yep, easily 250 MB/s off the Drobo.

$ cd
$ dd if=/dev/zero of=zero bs=$(( 1024 * 1024 )) count=8

Yep, easily 200 MB/s to the SSD… wait, why did it drop to zero after the first GB? What are all these spikes?!

*poke about system diagnostics*

Ah, TRIM is disabled. Probably a while ago, too. (I guess Apple still doesn't trust third-party SSDs to implement it properly?) Whatever. I now have two problems: I need to OSX to send ATA TRIM commands going forwards, and I need to TRIM all the current empty space in order to restore my SSD's write performance.

I open up Trim Enabler and slide the switch. Done. I go to the Settings tab, check "Warn if trim patch is disabled", so maybe I'll notice if this breaks again. Reboot, and System Information now says TRIM is enabled.

Now, how do I re-TRIM all the current free space? I could write a file of zeroes and delete it, but that's… dumb.

*poke about the web*

Ah, great! fsck can do this. Though… since I need to use my now-modified kernel extensions, I'll have to fsck my Macintosh HD volume while running from it. This can be done from single user mode, but asking a running system to modify itself in this manner is mildly unnerving. I'd feel better if I were booted from the recovery partition, but that would not include the modifications necessary to enable TRIM on non-Apple SSDs.

Before going further, I asked Disk Utility to "Verify Drive" from the desktop to make sure everything was fine.

Guess what? Everything wasn't fine. Apparently somewhere along the line my volume bitmap got screwed up; space was marked as used even though the filesystem no longer referenced it. That doesn't affect any of my data, but it does indicate that the volume somehow got in an inconsistent state, which means it's possible there are other undetected problems. Now the situation demands even more caution.

Time Machine! Back up everything!

Time Machine dutifully backed up everything, both to my NAS as usual and to an external hard drive that I infrequently attach.

I powered down, then powered up again while holding ⌘S. Once booted:

# fsck
...
** Checking Journaled HFS Plus volume.
** Checking extents overflow file.
** Checking catalog file.
** Checking multi-linked files.
** Checking catalog hierarchy.
** Checking extended attributes file.
** Checking multi-linked directories.
** Checking volume bitmap.
   Volume bitmap needs minor repair for orphaned blocks
** Checking volume information.
   Invalid volume free block count
   (It should be 82404658 instead of 81741301)
** Repairing volume.
** Rechecking volume.
** Checking Journaled HFS Plus volume.
** Checking extents overflow file.
** Checking catalog file.
** Checking multi-linked files.
** Checking catalog hierarchy.
** Checking extended attributes file.
** Checking multi-linked directories.
** Checking volume bitmap.
** Checking volume information.
** Trimming unused blocks.
** The volume Macintosh HD appears to be OK.

Hooray! It says the volume is fixed, and it issued TRIM commands for all the unused space. We should be in business.

# reboot

Now I see a globe instead of the Apple logo. Are you… are you trying to netboot? What? Why?

Did I do a bad?

Power down. Power up. Apple logo, flicker, Apple logo, login screen. (Whew.)

And finally: I tried copying that 7 GB file again. It completed in about 30 seconds, averaging 220 MB/s.

$ mongodump --help
Export MongoDB data to BSON files.

options:
...
 -o [ --out ] arg (=dump) output directory or "-" for stdout

Great, -o - is how I write to standard output. I want to dump from here and restore to there, so mongodump here | mongorestore there is the natural solution.

How do I restore from standard input?

$ mongorestore --help
usage: mongorestore [options] [directory or filename to restore from]
...

Hmm, well mongodump understood -, surely mongorestore will too:

$ mongorestore -
connected to: 127.0.0.1
don't know what to do with file [-]

Okay, so mongorestore thinks - is a literal filename. The example input path is in brackets indicating that it's optional; maybe it uses standard input by default if I leave it blank?

$ mongorestore 
connected to: 127.0.0.1
don't know what to do with file [dump]

All right, I give up. Google, how do I make mongorestore read from standard input?

Answer:

• SERVER-4345: mongorestore has no way to accept input from stdin
• SERVER-3111: Allow mongorestore to read from stdin

That's right! mongodump can write to standard output, but mongorestore cannot read from standard input. Reported two years ago, still unfixed.

You can of course dump to a file and then read from a file, and that's what I ended up doing. But… if that's the only permitted mode of operation, why support writing to standard output at all? It's just a useless and frustrating half-feature.

I poked around the MongoDB bug tracker to get a list of other "planned but not scheduled" tickets. As of this writing, an efficient representation of keys (#863), a read-only mode (#563), triggers (#124), and re-adding master-master replication (#2956) – each of them a feature I wanted – remain in that category.

For the record, Postgres95 beta 0.02 (released May 1995) supported pg_dump | psql.

After lots of head-scratching, I was able to determine that the users' local DNS is in fact broken and does not return SRV records.

For example, take this query against Google's public DNS:

; <<>> DiG 9.8.5-P1 <<>> srv _xmpp-client._tcp.google.com @8.8.8.8
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 10142
;; flags: qr rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;_xmpp-client._tcp.google.com.    IN  SRV

;; ANSWER SECTION:
_xmpp-client._tcp.google.com. 900 IN    SRV 20 0 5222 alt2.xmpp.l.google.com.
_xmpp-client._tcp.google.com. 900 IN    SRV 20 0 5222 alt3.xmpp.l.google.com.
_xmpp-client._tcp.google.com. 900 IN    SRV 20 0 5222 alt1.xmpp.l.google.com.
_xmpp-client._tcp.google.com. 900 IN    SRV 20 0 5222 alt4.xmpp.l.google.com.
_xmpp-client._tcp.google.com. 900 IN    SRV 5 0 5222 xmpp.l.google.com.

;; Query time: 36 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Fri Nov 01 09:42:32 CDT 2013
;; MSG SIZE  rcvd: 251

Running against Level3 DNS (4.2.2.2) and OpenDNS (208.67.222.222) return the same results. There are SRV records to be found.

Here's the same query run by an affected user with their default settings as installed by Comcast:

; <<>> DiG 9.8.5-P1 <<>> srv _xmpp-client._tcp.google.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 31875
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;_xmpp-client._tcp.google.com.    IN  SRV

;; Query time: 5 msec
;; SERVER: 172.16.12.1#53(172.16.12.53)
;; WHEN: Fri Nov 01 07:22:43 CDT 2013
;; MSG SIZE  rcvd: 46

The SRV records missing, but it doesn't stop there. Nonexistent names should come back as NXDOMAIN, not NOERROR. What's worse, the user's local DNS server has the balls to set the AA "Authoritative Answer" bit, something that only the actual google.com DNS servers can claim. And of course, to top off the wrongness, the device claiming to be authoritative leaves the authority section blank.

The hardware at fault is a Comcast-provided router, a Netgear WNR1000v2-VC. I don't know who wrote their DNS software, but it's totally and completely wrong. (Why can't they use dnsmasq or dnscache?) A vaguely-related Netgear device got a firmware update some three years ago that corrected a similar issue, yet the issue persists here.

My selected workaround is for the affected users to stop using their broken devices for DNS. Google public DNS and OpenDNS are both viable options, and both handle queries properly.

Serial communication boils down to bits – ones and zeroes – transmitting high and low voltages on one wire, receiving high and low voltages on a second wire. It's an extremely common way to interface with peripherals because it's extremely simple.

At least, it's simple until you get into the details.

Signalling details

How long do you send each bit before sending the next bit? (This is usually specified in terms of a baud rate which is related to but not exactly equal to this value.) How many ones and zeroes are sent per character? Are you transmitting extra bit(s) for error correction purposes, and if so, how are they calculated? How do you mark the start of a character? What about the end?

Familiar configuration strings like 9600,8,N,1 can answer few questions, but that's not the end of it. See… besides data, devices often need to communicate additional information.

You might want to reset the device somehow. Why not just hold the transmit line high (sending a constant stream of ones) for the length of several characters? This is the "break" sequence; it's not data (it's encoded in a way that cannot be confused as data), but it's transmitted on the data lines.

Another such need has to do with buffering, wherein devices can't accept more data until they've processed data they already have. A receiving device can:

1. Hope no new data arrives. What if it does? Well… throw it away, I guess.
2. Send a special "don't send me more data right now" character. This would be with the XOFF character… but now you have to worry about what to do if the XOFF bit sequence legitimately appears in the data stream.
3. Use a separate electrical connection to indicate that whether or not it's ready. This would be the CTS line… but now instead of two wires, you have three.

There's other signalling needs too. The host might want to know if there's a device plugged in (the DSR line), and the host might want to tell the device that it is plugged in (the DTR line). The host might want to tell the device that it wants to send data (the RTS line). The device might be acting as a bridge between the serial link and something else – say, a phone line or radio link – and might need to indicate that it has a connection to the other side (CD) or that something on the other side is trying to establish a connection (RI).

All this sucks, of course, because by now you've either ballooned from 2 wires to 9 wires (DB-9) to 25 wires (full RS-232) to handle everything or you've thrown up your hands in frustration and completely given up on trying to communicate most of this information.

Implementation details

It's woefully inefficient to force general-purpose processors to sit there and transmit and receive individual bits, so instead, people make dedicated hardware called UARTs – universal asynchronous receiver/transmitters – to do this.

The key advantage of a UART is that you can send and receive whole characters at your convenience. If you want to send a character, give it to the UART. If you want to send more, check to see if it's done, then send more. When it receives a character, it'll say so, and you can process it as a unit.

Of course, character-at-a-time operation isn't hugely more efficient than bit-at-a-time, so the new hotness 20 years ago was the 16550 UART which had onboard buffers. You could send and receive up to 16 characters at a time! Amazing!

Nowadays, 16550-like functionality is included in larger I/O chips alongside real-time clocks and other decidedly unsexy functions. But… remember how data isn't the only consideration? Yeah, well, UART buffers only deal with data.

If you want to send a break, well, poke at a register saying to send a break. If you want to signal things or receive signals on all those status lines, poke at another register, or check a register to see what lines are set. Catch is, since UARTs are asynchronous and include a buffer by design, it's very difficult to line up "I received this character" with "that status line changed"; you can be sure both happened, but rarely are you quite sure in which order.

Also, good luck changing baud rates or flow control settings at a specific point in a data stream. Those internal buffers? Not so great if you have a need for precision at the electrical level.

Unix

Serial ports are character devices on UNIXy systems, which means they look just like a file for reading or writing. Simple!

Of course, if you want to do anything with it, you need to specify the serial port configuration. That sounds like an I/O control, right? So, like, ioctl()?

Well… kinda. Serial ports and UNIX go way back – back to a time when people primarily used computers through teletype machines – so a lot of serial functionality is mixed in with terminal functionality. Terminals have a separate set of control functions (see <termios.h>) that are may or may not be implemented with ioctl() under the hood. Still, even with these functions specifically aimed at serial control, setting 9600,8,N,1 is far from trivial.

Sending and receiving data works like any other character device. Well, mostly. The operating system may or many not buffer things in ways you'd expect, and your data likely flows through a UART that may or may not buffer things in ways you'd expect, but it works.

What about all that other stuff? Like, how would you send a break? Call tcsendbreak()! What about receiving a break? Well, if the file handle in question is not your controlling terminal, you can ask to get breaks delivered as \x00 or \xff\x00\x00, your preference! But, wait… how would you distinguish breaks from normal data you might read with those patterns? You can't. Have fun!

How would you wait for data? Easy, use any of the normal ways you'd wait for data on a file descriptor. How would you wait for a status line change? Surprise, you can't! Poll ioctl(fd, TIOCMGET) several times a second and watch for changes yourself. Let's not even worry about detecting parity errors, truncated characters, or things like that.

See? Simple.

TCP/IP

It gets worse when people try to expose serial devices over a network, because most of the time, people just pump characters back and forth. Simple!

This might even work, but it's extremely naïve: all that other stuff is just glossed over, and anyone that needs a serial port to support anything besides transmitting and receiving characters will be very disappointed.

Enter RFC 2217. This is a specification that extends Telnet to include serial port controls. (Some people use Telnet as a dumb pipe for characters, but in fact it includes an extension mechanism.) Hooray! Now our serial port is back to being a single byte channel, there's a clear escape mechanism to distinguish control signals from data, and all the 9-wire serial port functions are available.

(Okay, it's really only almost all of those functions. For example, the UNIX API allows you to transmit with one baud rate and receive with another, while RFC 2217 does not. Also, while you can clear the buffers, you can't synchronize command execution with flushing the buffers, but in practice anything you need to do like that would suck using a local ports anyway.)

Programs such as ser2net can be used to expose physical serial ports as intelligent, functional RFC 2217 services. Perfect.

…unless you want to use software that assumes you have an actual serial port, or software that assumes a physical serial device. Aagh!

Google Earth says the photos were taken on Monday September 9, and judging by the shadows, it appears to be shortly after solar noon. I checked my logbook and found that I made a jump over lunch that day.

Curious, I dug a little deeper, breaking out my FlySight GPS logs. I usually run my FlySight for the entire ascent when I'm not concerned about battery life, and sure enough, I found I had recorded the plane's flight path as part of my jump #705 track. Following along, I find:

Hooray, data!

Here's a link to this view in Google Maps just for good measure.