I gave a talk on this subject to the first ever Code Fellows class, in Seattle last Friday. Here’s the slide deck, hope you like it.
Years ago, before great services like Dropbox were available, I purchased a NAS device for my home network. If you aren’t familiar, the idea is essentially to hook up a storage device that’s available while you’re at home, so that
I set up my NAS with two 500G drives, and proceeded to copy over my 20-year old CD collection (in MP3 format), all of my photos, and a lot of other important data. I have multiple computers, on various operating systems, in various rooms at my house, and they could all access my music and photos over both wireless and wired connections – it worked pretty well for a while.
Then one day, one of my drives went bad. I got on Amazon, searched for one of the drives on the very specific compatibility list the vendor published, found, purchased and installed it. It worked. I hot-swapped the drive for the bad one and everything continued to work great.
Being a generally paranoid person, and having experienced pains in this arena before, I also wanted all of my stuff backed-up somewhere outside of my house. So I shopped around, and decided to use the Memopal service. There were a number of nascent services coming on to the market at the time, and theirs seemed the most reputable and solid. I set things up so that the Memopal software was synchronizing files from my NAS to their cloud, set the account to auto-renew yearly, and for the most part left it alone. There were a few times when I checked on it, or when I had to fuss with their software to get it to work again, but otherwise it seemed to work pretty well.
Years went by, and in the last three years I got more and more busy with the start-up I helped co-found, BigDoor. My personal email queue was the last thing on the list to maintain, unfortunately.
A couple of weeks back, a very unlikely event occurred: both of the 500G NAS disks had problems. The manufacturer of the device, Infrant, was purchased by Netgear in 2007. I’ve been working with their support on this issue, and have high confidence that I’ll be able to recover my data after spending more money. Their support has been pretty good so far, and I’ll report back here if it goes south.
On the flip side, when I contacted Memopal support, I learned that even though they’d taken my money last year and this year via auto-renew, since I didn’t enter the licence code into their software last year, all of my data was deleted and can’t be recovered. This approach is new to me; I’m used to the standard “if you pay for something, we’ll at the very least not destroy what you’ve paid for” vs. “to prevent the irreparable deletion of all your data, it’s not enough just to pay us, you have to put the code we emailed you into our software”. Below is the support thread, in case you’re as incredulous as I continue to be.
With product decisions like this, it’s no wonder they’re getting their asses kicked by Dropbox and other new services. I’m curious to hear if you can think of a good reason why an online data back-up service would collect payment, but then delete your data (without reasonable warning), because you didn’t enter a licence code.
2/1/2013 Update : fairness in reporting; below is how their customer service responded. Amazing.
This was originally posted last week here, but I wanted to re-post here and will be updating with our latest status and learnings, if there’s any interest. Let me know.
Ever since we registered our startup’s domain with them years ago, we’ve been anxious to get off the free DNS provided by GoDaddy at the least, and ideally change registrars as well. With all the issues other companies have had with them + their political positioning … we just want out. It’s actually embarrassing to admit we were in this situation for so long, but I’m swallowing my pride in hopes that this will help others out – open-sourced embarrassment (O-ASSMENT). Until recent, we really haven’t had the time/resources to tackle it without affecting product development efforts and higher priorities. One of our senior guys has been exploring options for weeks, and we thought we were in a good position to make a change.
There are two parts of this puzzle that need to be fit: GoDaddy is (was) the registrar of our domain, and they also are hosting DNS for us. That’s a typical set-up when you first register your domain these days; most registrars also offer managed DNS. But it’s not a good practice to leave your DNS hosted with your registrar – it’s better to separate them right when you register the domain, if you can.
The two parts (registrar and managed DNS) are intertwined; I’m trying to avoid DNS details for the non-technical, but essentially/simply/horribly put: DNS is much like a big phone book that yourDomain.com has a page in, that page maps IP addresses to friendly names like http://www.yourDomain.com and api.yourDomain.com. One particularly critical mapping provides the IPs pointing to our authoritative name servers. This mapping is also stored in the index of the phone book, by a higher DNS authority…like Elvis. Servers that need to know where http://www.yourDomain.com is (in other words, its IP address) look in the index if they need to, and then get the IP from our page in the book. This is where the registrar comes in – you can only change the IP of the authoritative name servers through the registrar of the domain. Otherwise, with regard to DNS/WHOIS records, the registrar is just a text string, a name without a number.
But this makes registrars ultimately all-powerful; you can make all the DNS changes you want, but if the authoritative name servers are changed and pointed to hosts that don’t have our DNS information, or don’t have the right information - you’re totally FUBARD.
We shopped around for a different registrar, and at one point were ready to sign an expensive deal with MarkMonitor, who from all accounts is the market leader in terms of locking things down from a security standpoint. But they couldn’t seem to get their act together fast enough and were too expensive for our growth stage anyway. We decided to go with NetworkSolutions, the “first” registry operator and registrar for the com, net, and org registries.
GoDaddy offers free DNS when you register your domain with them, but they also offer Premium DNS. We upgraded to premium weeks ago, to get a better idea for our DNS traffic and to price out competitors. To be totally clear, at this point in the story we’re paying GoDaddy for their premium DNS hosting option. GoDaddy offers this to their customers as a stand-alone service; in other words, you can use GoDaddy just as a managed DNS provider (as long as you have a domain or two registered with them, I’d assume ).
So, given that we wanted to move our registrar (because we didn’t want GoDaddy to own the gate to our authoritative name servers), and our DNS, we had a few options:
Here’s the relative timeline, what happened, and what we expected/should have happened:
Fortunately our end-user-facing product fails gracefully under these circumstances and customer impact was minimal.
That whole escapade pretty much escalated the priority of us getting off their managed DNS, which we did in the next week. After looking at various (mostly expensive) options, we moved over to Amazon’s AWS Route53, which went relatively seamlessly. The nice thing about Route53 is that it’s accessible programmatically and can be managed via scripts just like the rest of our AWS resources.
I totally get that the herky-jerky that comes with WHOIS-on-first; name server and DNS transfer of ownership puts registrars in an odd situation, one that requires competitors to coordinate if they’re going to act in the best interest of their soon-to-be/just-cancelled customers. But there’s got to be a better way than this ridiculous bullsh** we just went through. Registrars who offer DNS hosting as a service have an obligation to publish the ‘how do I get out without getting ass-f*****’ instructions at the very least. Better yet, for a grace period, leave DNS the way it is until an NS record gets changed at the root level, messaging their customers about what’s coming in the meanwhile. I know that some registrars do provide a grace period like this.
I’m obviously not a registrar, and admit that my proposed solutions may not be tenable. But there’s got to be a better way.
We’re not the only startup in the bus that’s running over GoDaddy, there’s pretty much wide agreement on this topic. I’m glad we’re over that speed-bump and the startup bus is barreling forward at high-speed as usual.
I’m tempted to turn this into an ICANN complaint – any input on whether that would hold up, or be worthwhile? ( to comment you have to be on this post’s page, rather than the blog home page)
Update : in case it’s helpful for anyone, I’ve started gathering some numbers on what some other friend’s startups are using (without major complaint) for registrar and hosted DNS, and will update here for now. Please email me directly if you’d like me to add something to this list.
This morning before our board meeting (I guess it was officially yesterday morning, now) I had the wonderful good fortune of giving my friend Andy a big congratulatory hug, in celebration of his beating cancer – he had just gotten the confirmation an hour or two before. Fantastic. Incredible, for so many reasons. I’m thankful and happy for him.
Hours later after leaving the office late I was still working in bed, typically banging away at email on my phone around 2:30 AM, when I got an email from a girlfriend I had in college, a very close friend who I hadn’t talked to in years. She thought I already knew, but it came as a total shock : that’s how I found out that one of my best college friends, roommate and co-conspirator in mischief of all kinds, had passed away in February at the young age of 43. I was of completely stunned.
The first thing I did was start typing his name into Google on my phone, which started auto-completing before I could finish his last name:
My heart sank when I saw the word ‘cancer’, and my mind went into an odd state of surreal incredulity. I got out of bed, put a sweatshirt on, and went to sit in front of the Mac in my studio. I did the same Google search, which right away produced a full-page of results specifically about my friend Adam Adamowicz, headlined by a Wikipedia entry. Just now, wondering how deep the relevant results went, I opened page 13 of the results and it was still a full-page of links to sites discussing and mourning his death.
I opened all of the search result links on that first page, and started looking through the pages – then my mac crashed. Already straining to see clearly through teary eyes, now swearing at this inanimate object that was acting the proxy for my dead friend, I moved to a PC laptop which was sitting on the floor and started again.
The New Funeral for old out of touch friends in the virtual age. Strange but wonderful way to say goodbye. Really, I feel lucky to be able to do it.
Clearly it was easy for people who loved Adam to paint their version of the artist. This page and the quotes from his friends really nails it. Here’s a link to the New York Times article about him. Awesome Robo created a wonderful post, if you get a chance to view the video on that page you’ll see him at 1:29, sitting among his vast creation. The Reddit post about him is long. It looks like his artwork is on display at the Smithsonian. It goes on and on, he touched so many people – truly amazing and wonderful.
I first met Adam when I was a freshman in college at CU Boulder. I’d been assigned to an unusual room on the basement floor, in the corner of the building, with two other guys. There was a door on the back wall of the room, which I discovered with great happiness could be opened with the appropriate amount of cajoling, leading to a room roughly an eighth of the size of our regular dorm room. It was totally over-ridden with dust and spider webs, pipes running everywhere – I fell in love with it instantly, moving all of my loaned CU furniture out of its clean and safe haven to the dark windowless hovel that became my home for the next two years (I successfully petitioned to stay in the same room the next year). One roommate hated me, the other was amused.
One of my roommates was the quiet, introverted, artsy music-head type of guy that I could totally get along with. I think he was impressed with my obsession with living in a dank closet-like hovel and making odd senseless contraptions, and that’s what likely led to him introducing Adam and I. That first year for Halloween, after painting his face a gruesome green and donning his black leather biker jacket that we bike-less artist types consistently wore those days, my roommate took me to the next dorm over to meet him. When we walked into his room, (probably Skinny Puppy blaring), I recall being instantly wowed by the costume makeup he had self-administered. It was an odd mix of hollywood horror steam punk, and it garnered instant respect.
Those costumes got more and more amazing every year.
I remember taping cheap manila paper up on the walls of my hovel, buying a bunch of pens, and luring Adam over with a 12-pack to get him to draw art on my walls. He got wise to that, but not until after I got a few drawings and after he’d laughed his ass off at my ‘thinking chair’ (my loaned CU lounger up on cinder blocks, with an attached clothes-hanger frame suspending a beer-can mobile powered by a fan motor which caused Bud aluminum to spin around your head at high speed). I remember the beer-ish night we were walking around campus and I introduced backwards-man to him, who comes out as my alter ego sometimes after putting foot in mouth; I did a full audio and physical rewind that made him roar and take up.
My buddy Joel and I got our own place the next year. We didn’t have enough money for a real apartment, so we rented the attic of a house.
The fascist owner of the house had turned the attic into a lie of an apartment, a rubes cheap trick that we had to stoop to walk around in unless we were walking the center line. We were glad to have a cheap roof on our heads. One person was cramped in that hot cubby, two people were constantly bumping into each other. Adam needed a place to stay, so of course he started staying with us. There were only two beds, divided by a make-shift wall made out of unwanted propped-up doors and cardboard, so we took turns sleeping on the floor.
I’d just discovered audio sampling, and had bought a new keyboard, so there were many drunken nights of tempo-shifted bodily noise hilarity. We had a scorpion living in a gutted TV set, and Joel had his homemade weight-lifting rack up there in the attic, for those late-night bench-press contests that our neighbors loved. I’d always wanted a gargoyle at my castle’s entrance, so I made one out of a motion detector, some duct tape and broomsticks (fail). Adam was drawing for comic books and he had me pose in my overcoat for a cover of 2,000 maniacs, brandishing a knife. I later framed it, it’s in my master bath and I look at it every day – so he’s never quite left me.
Then Adam brilliantly found an incredible warehouse in north Boulder to live in, right next to the strip club, auto shops and welding studios. Here he is in a junk car in his backyard, back then.
The warehouse was perfect for him, and he turned it into an artist’s wonder home. He had fantastic parties there; I remember lighting Joel’s pant leg on fire with lighter fluid, him returning the favor (pre-planned of course, we wore layers), golfing off the roofs of junked cars, and how he took a saws-all and cut the roof off of his already debilitated Saab because he wanted a convertible. At one point he was making great money creating art for raves; crazy huge foam stuff that hung from the ceiling for the trippers.
He worked as a bartender at a local nightclub downtown; friends, brother and I would go and visit and play pool past closing time. Some weekend nights we’d go to Denver, and Adam and I climbed a fair number of fire escapes to be on the roofs of buildings in the middle of the night, with not a small amount of beer driving us skyward. Adam was full of fire, an adventurer. He was the kind of guy you always wanted to hang out with, it was always fun.
I lived in Denver for a while, and these shots were supposed to tell a story, a ludicrous art collaboration, something along the lines of:
“Whoah, that’s a pretty tall tree, Adam. Think you could climb it, and lower yourself on a rope, with a 10-speed bike?”
“Hmmm….let’s seeeeeee here…”
“Ha ha! I was hiding in a large pile of leaves with the hose, to shoot you down!”
Then it came time to move away from Colorado, him to San Fran and I to Seattle. One one of the various trips back and forth to the coast, we packed his stuff into a cargo van I had at the time and drove west, stopping at Lake Powell for an awesome time. This is an amazing place that you really should check out sometime.
I dropped him off in San Fran and headed to Seattle to start a new life.
I’ve never known anyone to write letters regularly, except my brother and Adam. And he wrote hilarious, entertaining, endearing letters. It may sound odd, but I couldn’t throw them away – I’ve kept them in a box with other important personal stuff I’ve collected over the years. They’re that great.
After a few more years we’d largely lost touch, and even a significant falling out (my fault) , but he continued and still continues to remain on my short list of all-time favorite people ever.
I wrote this largely for myself, to record all of the memories flooding back in an instant after hearing the word. But I also wanted to share my perspective on the man who left us, and I hope you find it as entertaining as the memories I have of our friendship.
I am entirely shocked, saddened, and full of regret. My heart goes out to all the people who were close to him and loved him, to our mutual friends wherever you are, and most of all to his family.
Remarkably, one of the last things I said (wrote) to him was “I love you, man”, via email back in March of 2010. Lucky.
Adam, you made the world a better place.
April 26, 2012
Fuck cancer with a flaming barge pole.
Wow, since I blogged last I chose Python/Django and we rocked the gamification world with it. Lots of stuff I should have been chronicling. Trying to get back to it…but there’s a single event that prompted the return, I’ll be posting shortly on that.
I’ve been learning the basics of these programming languages and development platforms lately. I’m impressed with the huge Rails movement, and Django seems like a very cool project. Both platforms allow web sites to be built at an incredible pace.
Both Ruby and Python appear to be relatively straightforward. But I must say that at this still-early stage in my learning, Python wins for simplicity and ease. Ruby’s “code blocks” are a bit obtuse, as example reasoning.
From what I currently understand about Rails and Django, their strengths lie in auto-generated code whether the implementation be application-supporting code derived from database schema (Rails) or database schema derived from application code (Django). Microsoft’s LINQ, which we’ve been using a bit at work, is similar but not foundational to the platform. It’s all cool stuff that makes DBAs anxious, which is always fun.
After trying to test out Google’s AdWords client API using their supplied code samples, finding that those code samples require old libraries that the latest Python release doesn’t support, and realizing that Django itself won’t support the latest version of Python (3.0) for a year or more to come, I’m wondering what it would be like to have a wonderfully supported and rich platform like Rails tooled with a simple and evolved language like Python.
Now I’m wondering what I’m going to have for dinner. I’ll probably get more mileage out of this wonderment.
A number of years ago I was employed as an audio technician, and had the wonderful opportunity to travel around the country, recording people say interesting things. Out of all the events that I got to attend, TED was my absolute favorite. I’d come home boiling over with inspiration and enthusiasm for technology. If you aren’t familiar with it, check out some of the video that they’ve graciously made available for free. I just watched a very cool clip with Pattie Maes and Pranav Mistry about MIT’s “Sixth Sense”.
I repaired an Onkyo DV-CP702 DVD changer today. Below is a picture of it with the case still open, playing “This Film Is Not Yet Rated”.
The DVD player had stopped working months ago. You could hear it loading discs, but nothing would play.
Here’s how I fixed it:
First, I unplugged it. This was a key step.
Then I removed the outer panel by unscrewing the six Phillips machine screws and tilting it up from the rear.
Then I sat in wonderment for a good fifteen minutes, gazing at the complex beauty inside. Gears, motors, circuit boards, a fuse that hadn’t blown, a dead spider.
What to do next? It wasn’t very interesting without electricity flowing through it. I noticed that there was a plastic shield covering what appeared to be the most interesting bits. It had a big yellow warning sign on it that said something to the effect of “Watch out, there’s a laser in here. Don’t look into it.” Miffed that the innards of this remarkable piece of equipment were causing me to read, I removed the shield.
Again, I detected no movement, nor anything of any interest happening whatsoever. So, of course I plugged it in. Still nothing.
Then I turned it on.
A few things happened at that point: it went through a boot-up process, and then the carousel turned, trying to load disks. The carousel is a round plastic disc that holds the six DVDs or CDs. It turned one time for each slot, stopping to allow the disc-reading mechanism to pop upwards and cradle a loaded disc. I had put one in there, and when it got to my disc the disc-reader popped up under it, elevating it slightly. I heard the mechanism that moves the laser engage, and then…nothing.
So I stared at it for a while. Why no workie? Finally it struck me - the disc is supposed to spin. Why disc no spin? Hmm. Time to take more apart.
Unplug. Unscrew disc-loading mechanism, unplug electrical leads. Look at mechanism more closely. I noticed there were two motors: one for moving the laser back and forth, and another for turning the loaded disc. The disc-turning motor had two small screws holding it in the chassis, and two wires coming off of it. The spindle was crowned with the smaller disc that the DVD sits upon, and after looking without success for a tension screw, realized it had to be leveraged off by force with two screwdrivers. I then unscrewed the motor and clipped the wires where they connected to the circuit board.
Then I figured I better get an electrical parts manual, to find out what part number I needed to replace. Ten dollars later, I had found the repair manual for the DVD changer on-line and had identified the part number needed, but didn’t have a lead on where to find one. I later realized that it would be easier and cheaper to read the part number right off the motor itself. Brilliant.
I Googled the part number, and found Mat Electronics, an on-line distributor who had one in stock. Motor: $4. Shipping: $7. I ordered it. Then I went on vacation for two weeks.
When I got back from vacation, I had completely forgotten where I was in the process, and couldn’t find any evidence (including a delivered motor) that I had ordered it. After looking at the cost, and realizing that I’d have to buy a new soldering iron to finish the job, I decided it wasn’t worth my time. It was going to likely cost me $50 bucks to fix a $200 DVD changer using a motor that was listed as a “high failure” part. Screw it.
A week later the part arrived, much to my confusion.
Time to go pick up a soldering iron at Fry’s: $6.
So I brought the soldering iron home and got to work. I screwed the motor back into the chassis, and soldered the two wires back onto the circuit board. I then pressed the seat back on the spindle, pushing it all the way down. I replaced the chassis and dreaded tiny spring, and used my soldering iron to re-glue the spring to the chassis. Then I screwed the shield back on.
Excited to see my work in action, I plugged it in and loaded a disc. Disc no spin. Nothing. Bummer. Oh well, it was a fun little project. Fail fast and early to save time, cost and headache.
Nope – I couldn’t let it go. I stared at it for a while. Then I removed the shield and put a piece of tape over the laser to protect myself from my own morbid curiosity. Load another disc. Disc no spin. Closer look. Spindle seat was pressing against the laser housing! Oh my goodness, that can’t be good. I had pressed the spindle seat too far down on the motor spindle. Dang me, blast it.
So I pried it off the spindle a bit and tried again. Nothing.
Time to grasp at straws. The manual shows how to reset the micro-controller, so I did that. Still nothing. Closer look – the spindle seat was still pushed pretty far down. Pry it off a bit more.
Then – it happened. After plugging it back in, the disc spun!
I brought it out of the shop and into the nice warm house where I plugged the video out into a TV. The disc spun, but was making a strange noise. Not like “whirrrrr”, but more like “brrrrrrgrrrrwhrrrrrbraaaa”. The player readout said “No Play”, which typically means a dirty disc. I’m sure you can imagine the problem, given that description. The spindle seat was still pushed too far down, and the disc was moving vertically up and down too much. Unplug, dismantle, adjust spindle seat again. Clean laser lens, just in case. Reassemble, plug back in and…voila! Taking the photo above was the next thing I did after jumping up and down a few times and crying out like a schoolgirl.
The total cost without the manual was actually under $20, and I spent a good three or four hours taking things apart, driving for parts, and reassembling them.
I make a comfortable living in the software business, and my time is very valuable to me. Was all this time, effort and money spent to fix a $200 outdated and prone-to-fail-again electrical component worth it? Hell yes.
Why not just buy a new one, or have someone else fix it (outsource it)? Because it was fun.