TL;DR — Summary of the Problem

Using a text editor to modify content management system (CMS) configuration files (like wp-config.php) could expose your database password to the world. Several popular text editors like Vim and Emacs automatically create backup copies of the files you edit, giving them names like “wp-config.php~” and “#wp-config.php#”. If the text editor crashes or the SSH connection drops during editing, then the temporary backup files may not be cleaned up correctly. This means that the CMS config file (which contains the database password) could accidentally be made public to anyone who knows where to look.

Most servers, including the ubiquitous Apache, will happily serve the plaintext of .php~ and .php# files without passing them through the PHP preprocessor first, since they don’t have the .php file extension. Thus, your sensitive database credentials are just one GET request away from being accessed by a malicious party.

I wrote an automatic program, which I call CMSploit, to test for the prevalence of this issue across the wider web. I tested the top 200,000 websites (as ranked by Quantcast) and found that 0.11% of websites are vulnerable. If we eliminate non-CMS sites, and just look at CMS-powered websites, then we find that 0.77% of websites running a CMS have publicly-visible config files.

If you want all the gory details, then keep reading.

CMS Configuration Files

Most content management systems (CMSs) store sensitive settings like the database hostname, database name, database username, and database password in a file that sits in the root of the web directory.

Here is what a typical config file looks like:

<?php
define('DB_NAME', 'my_secret_database'); /** Name of WordPress database */
define('DB_USER', 'secret_agent_1'); /** MySQL database username */
define('DB_PASSWORD', 'you_will_never_guess_this'); /** MySQL database password */
define('DB_HOST', 'localhost'); /** MySQL hostname */

Here is a list of the various configuration files used by the most popular CMSs:

wp-config.php        # WordPress
config.php           # phpBB, ExpressionEngine
configuration.php    # Joomla
LocalSettings.php    # MediaWiki
mt-config.cgi        # Movable Type
settings.php         # Drupal

Despite the fact that these configuration files exist in a publicly accessible folder, the file contents are unviewable by a normal web user. Accessing the file directly does not work because the PHP interpreter handles all requests to .php files and wisely returns a blank page instead of the actual file contents. (Try accessing http://www.example.com/wp-config.php on your favorite WordPress blog to see what I mean. You should get a blank page back.)

As you can see, all the sensitive database information is located within <?php ?> tags. So, even if a malicious user were to access your config file directly, the PHP preprocessor would just run the PHP code, which defines some PHP global variables and then returns a blank page. Thus, no harm done, right?

Text Editors Make Temporary Files

Popular command line text editors like Vim, Emacs, Gedit, and Nano create several temporary backup files during the course of file editing. When you open a file for editing, a backup of the original file is saved. Depending on your text editor, in-progress file changes might also be saved to a swap file, so you can restore your unsaved changes in the event of a program crash, power outage, or connectivity issue.

If all goes well, when you’re done editing the file, the text editor deletes the temporary files so your filesystem doesn’t accumulate dozens of old temporary files. However, if your text editor crashes or you lose your connection, then the temporary files will still be on your filesystem.

Here are the temporary filenames used by the most popular text editors (assuming a file named wp-config.php):

  wp-config.php~        # Vim, Gedit
 #wp-config.php#       # Emacs
 wp-config.php.save    # Nano
 wp-config.php.swp     # Vim (swap file)
 wp-config.php.swo     # Vim (swap file)

Putting Two and Two Together

If a CMS user edits a config file on their live site (as opposed to editing it offline and uploading it over FTP), then there may be temporary files which contain their database password floating around in publicly-accessible folders.

If someone requests one of these temp files, then most servers will return the plaintext, skipping the PHP parser completely — yikes! By default, Apache assumes that only files which have a .php file extension are PHP files. If the file extension is not .php, Apache happily serves up the plaintext of the file.

How prevalent is this problem?

After noticing this security issue on one of my websites, I became curious to find out how common it is across the wider web. So, I wrote a program to test the top websites and get a rough idea of the prevalence of this problem. I call it CMSploit. The program is pretty simple — it issues GET requests to a site to test for the presence of temporary backup files with common CMS config filenames.

Here were my results:

• Tested the 216,391 most popular websites (according to Quantcast).
• Found 230 config files visible in root of site.

• Thus, 230 / 216391 = 0.11% of all websites are vulnerable.

• Latest stats say that about 13.8% of the top 10,000 websites run CMSs. If we just focus on CMS-powered websites, then the percentage of vulnerable sites is much higher:

• Thus, 230 / (216391 * 0.138) = 0.77% of websites running a CMS are vulnerable.

It’s shocking to think that 0.77% of websites (1 out of every 130) built with a CMS has its database password just sitting there in a public folder for all the world to see. Lots of these are popular, active websites. You would likely recognize many of them. Most of the sites were WordPress blogs, but there were a surprising number of e-commerce sites too, which is a little scary.

Responsible Disclosure

1. I contacted several of the highest profile sites to notify them of this security issue on their site. Most of them fixed the problem within a few days. All who replied to me were extremely grateful for bringing the issue to their attention. One of the companies even offered me a free license to their software.

2. I submitted a vulnerability report with US-CERT. Unfortunately, they replied with “This issue is not the type of vulnerability class we are inclined to coordinate or publish on.” I also plan to submit reports with Apache, PHP, WordPress, and Vim/Emacs.

3. After running the script and collecting my research statistics (published above), I securely wiped all the config files from my hard disk. I did not attempt to login with any of the database credentials I discovered. Therefore, it was not possible to determine what percentage of the database credentials were valid or what percentage of database servers were open to remote connections.

Lessons Learned

1. CMS users should never edit their “config.php” file (or other sensitive files) with a text editor that creates temporary backup files. The best policy is to avoid editing any sensitive files on a live website. Instead, copy the file locally, make your edits to it, and copy it back to the server.

2. It’s trivially easy to write a script to search for vulnerable sites. Bad people have probably been doing it for several years. In fact, this issue has even been discussed in other forums before. You should check your sites for “wp-config.php~” and related files. Make sure your sites are not vulnerable.

3. Someone should fix this. It’s not completely clear where responsibility lies, though. Is it Apache’s fault? Or PHP? Or vim/emacs? Should WordPress and other CMSs do something about it? There are many ways to fix this problem. I don’t particularly care how it gets fixed, as long as the default configuration of Apache + PHP + vim/emacs + WordPress don’t have this problem, I’ll be satisfied. In the meantime, using this very common web stack we have a scenario where ~1% of sites expose their passwords. This is bad.

4. In the short term, you should proactively protect all your websites. If you run WordPress, you can block access to any file containing the string “wp-config.php” with a .htaccess rule like this:

   <Files ~ “(^#.*#|~|\.sw[op])$”>
   Order allow,deny
   Deny from all
   </Files>

5. You should configure MySQL to deny remote connections to your database and connect to localhost instead. If you absolutely need remote access, then explicitly whitelist certain IPs and deny the rest. This way, if someone gets your database credentials, they’ll be unable to actually log in.

Final Thoughts

Even though the discovery that 0.77% of CMS-powered websites have public database passwords is already shocking, I’m pretty confident that you could easily double or triple the number of vulnerable sites with a better, more thorough script, and lots more time.

The script I wrote only tests the root of each site for CMS config files. However, lots of sites run CMSs in subfolders and subdomains like /blog/, /wiki/, /forums/, blog.mydomain.com, etc. Testing these places would dramatically increase the number of vulnerable sites detected.

I will not publish the source code of this script, because of the potential for harm. However, if you are a security researcher and are interested in reviewing the source code, send me an email.

This DEFCON talk is relevant: Pillaging distributed version control system repos for fun and profit.

Discussion on reddit.

Slides from a presentation I gave to the Stanford ACM about CMSploit:

Thanks to John Hiesey for reading a draft of this.

Related posts:

1. HOW TO: Spy on the Webcams of Your Website Visitors

Update 10/20/2011: Whoa, this story is everywhere!

• CNET
• Wired.com
• The Register
• Ars Technica
• Gizmodo
• PC World
• Yahoo! News
• ZDNet (and another)
• The Inquirer
• Computer World
• The H Security
• An interesting opinion piece: “The Sins of the Flash”

Update 10/20/2011: Adobe says they just posted a fix to the Settings Manager that should resolve the issue. I just tested it out, and indeed the issue appears to be fixed now. Congrats, Adobe, for the quick fix!

Update 12/21/2011: This attack made it into Jeremiah Grossman’s list of top web hacking techniques of 2011. It’s #26.

Update 1/10/2012: Another similar clickjacking attack was just discovered and fixed by Adobe.

Original post:

I discovered a vulnerability in Adobe Flash that allows any website to turn on your webcam and microphone without your knowledge or consent to spy on you.

It works in all versions of Adobe Flash that I tested. I’ve confirmed that it works in the Firefox and Safari for Mac browsers. Use one of those if you check out the live demo. There’s a weird CSS opacity bug in most other browsers (Chrome for Mac and most browsers on Windows/Linux).

Video demo of the attack:

Source code: Github

Clickjacking + Adobe Flash = Sad Times!

This attack works by using a neat variation of the normal clickjacking technique that spammers and other bad people are using in the wild right now. For the uninitiated:

Clickjacking is a malicious technique of tricking Web users into revealing confidential information or taking control of their computer while clicking on seemingly innocuous web pages.
— Wikipedia

Combine clickjacking with the Adobe Flash Player Setting Manager page and you have a recipe for some sad times.

Background

I took a computer security class (Stanford’s CS 155) last quarter and really enjoyed this research paper on framebusting and clickjacking. After reading it, I checked out a few popular sites to see if it was possible to clickjack them. After a couple hours, I had no success.

But, then I stumbled upon this blog post entitled “Malicious camera spying using ClickJacking” where the author shows how to clickjack the Adobe Flash Settings Manager page to enable users’ webcams. He accomplishes this by putting the whole settings page into an iframe and making it invisible. Then, unsuspecting users play a little game and unwittingly enable their webcams. Adobe quickly added framebusting code to the Settings Manager page (why wasn’t it there in the first place?), and the attack stopped working.

But alas, the same attack is actually still possible.

How my attack works

Instead of iframing the whole settings page (which contains the framebusting code), I just iframe the settings SWF file. This let me bypass the framebusting JavaScript code, since we don’t load the whole page — just the remote .SWF file. I was really surprised to find out that this actually works!

I’ve seen a bunch of clickjacking attacks in the wild, but I’ve never seen any attacks where the attacker iframes a SWF file from a remote domain to clickjack it — let alone a .SWF file as important as one that controls access to your webcam and mic!

The problem here is the Flash Player Setting Manager, this inheritance from Macromedia might be the Flash Player security Achilles heel.

— Guy Aharonovsky

This is a screenshot of what the Settings Manager .SWF file looks like:

Live Demo

I built a quick proof-of-concept demo to show how it works.

Important: The demo is only guaranteed to work in Firefox and Safari for Mac. Right now, it doesn’t work in most other browsers since you can’t change the opacity or the z-index of an iframed swf file. However, I discovered a workaround that involves multiple iframes, but haven’t implemented it yet since it’s a bit complicated. But, I’m pretty sure that it’s possible to make it work everywhere, given enough time.

View the Demo.

The code is also available on Github.

I should also mention that my demo builds heavily off of the ideas and work done by the dude who runs this blog, Guy Aharonovsky.

Also: If you’re a bit leery about running the demo… I promise I’m not saving the webcam video. I just display it back to you so you can see that it works. However, if an attacker used this technique, they would almost certainly NOT show you any sign that your cam is on. You’re only hope of finding out that something’s up is your webcam indicator light (if you have one).

Why release this?

I reported this vulnerability to Adobe a few weeks ago through the Stanford Security Lab. It’s been a few weeks and I haven’t heard anything from Adobe yet. I think it’s worth sharing it with the world now, so that Adobe pays attention and fixes it more quickly.

Although every browser and OS is theoretically susceptible to this attack, the process to activate the webcam requires multiple highly targeted clicks, which is difficult for an attacker to pull off. I’m not sure how useful this technique would actually be in the wild, but I hope that Adobe fixes it soon so we don’t have to find out.

Further reading

If you want to learn more about clickjacking and framebusting, you should read the excellent Busting Frame Busting: a Study of Clickjacking Vulnerabilities on Popular Sites (PDF) paper by Gustav Rydstedt, Elie Bursztein, Dan Boneh, and Collin Jackson.

Related posts:

1. 1% of CMS-Powered Sites Expose Their Database Passwords

YouTube Instant made it onto the shortlist for Best API Use in the .Net Awards. It’s right there with Instagram and Gowalla!

I want to thank all the people who voted for me — I feel so honored and humbled by all the attention. I’ll do my best to keep making cool stuff that delights people.

Also, I want to thank the good people at .Net Magazine for putting on this awesome award series to recognize so many of the creative people who work in web design and development.

Related posts:

1. 2011 .Net Magazine Awards
2. YouTube Instant Around the World

In short:

I learned how to program by building lots of websites.

The full story:

I learned how to program by working on lots of different website projects starting from a pretty young age. What follows is a full account of all the major websites I’ve built, back to the very first site I made when I was 11 years old. What I hope the reader takes away from this full retelling is the importance of doing lots of side projects if you want to learn to program well.

The best way to learn a new skill is to practice, practice, practice. All the best programmers that I know sincerely enjoy programming — it’s something that makes them absurdly happy to do. And, so they do it a lot. Often, an unhealthy amount. Learning how to program — and how to do it well — doesn’t take superhuman ability. It just takes a willingness to get your hands dirty and build stuff.

It doesn’t matter what you build, as long as you pick something and start. The good programmers who I know each had a different reason for initially learning how to program. Some learned so they could make video games. Some learned so they could solve their own computer problems, or work more productively. Some learned so they could build products that make people happy. Some (the true hackers) learned programming as part of a larger goal of learning how computers work at a really deep level; they want to understand the machine. Some programmers just do it because they enjoy solving difficult problems.

The single factor that unifies all these types of “good programmers” is that they all got obsessed with programming at some point in their lives, and subsequently spent a long time programming. Lots and lots of side projects.

So, without further ado, here is the story of how I learned to program:

My first website

When I was like 11 or 12 years old, I decided I wanted to make a website for myself. I can’t remember exactly why I wanted a website, I just remember that I did. So, I searched the Internet for free information about how web pages, web browsers, and HTML worked. A lot of the information I found was out-of-date, plain wrong, or advocated bad practices (like making separate websites for Internet Explorer and Netscape), but it was really interesting and I learned a lot of neat stuff.

Despite the shoddy information I found online, I was able to make a simple website, which I called “Feross’s Website”. I built it with Microsoft Frontpage, which had really cool side-by-side WYSIWYG and HTML editors. I could make changes using familiar commands like Bold, Italics, etc. and see how that affected the HTML code in realtime. A great way to learn.

Here are some screenshots of my first site. It’s no longer online.

You can’t see it in the above screenshot, but almost every element on the site blinked, flashed, moved, or made sound. I put a different MIDI song on every page of the site. They all played automatically and there was no way to stop the music, unless you muted your speakers. Ah, good old web design from the early 00′s

As I got older, I tried to make my site better by redesigning it. I used free website templates that I found online and modified their images in Microsoft Paint.

Even though I built heavily on existing templates, I think this was a pretty good way to learn how HTML and web browsers worked. “Feross’s Website” didn’t have a purpose other than to collect a few movies I made as a kid, so it got boring after a while.

My first real project

In 9th grade (14 years old), my friends and I were pretty obsessed with watching flash movies and videos on websites like Newgrounds and eBaumsWorld (this was before 2005, so YouTube didn’t exist yet). I spent lots of time on these sites so I knew about all the best videos and games. I thought it would be really cool to make a website that collected all my favorite flash animations, videos, and games from around the Web in one place. That’s where I got the idea for FreeTheFlash.com. Here is what it looked like:

I used all the HTML I learned from working on “Feross’s Website” and also got my hands on a copy of Macromedia (now Adobe) Dreamweaver, which allowed me to use templates for the repetitive parts of the site.

After a while, I realized that I should make the site dynamic (I remember hearing that buzzword a lot), which basically meant that the site would be powered by a programming language like PHP, instead of just static HTML. So, I bought a book called PHP and MySQL for Dynamic Websites for $20 from Amazon and redesigned the site to use PHP and MySQL. I also gave it a fresh coat of paint:

I continued to work on FreeTheFlash for 2 years in high school. It was pretty successful for my first attempt at a “real” website — it had 600,000 visitors and 3 million page views in 2006. FreeTheFlash taught me how awesome it feels to make a product, stick it out there, and watch lots of people using it. It made me want to build a lot more websites.

My second website

In high school, I took pretty good notes for a few of my AP classes. So, in 11th grade, I decided to put my notes online for other students to use, if they didn’t feel like reading the textbook. I made a site called StudyNotes which I built with PHP and a content management system called Joomla. I also experimented around with Drupal, but found it to be too complicated.

That same year, I also made a website for my school’s Key Club chapter. It’s archived here.

During this period, I spent a large amount of my free time reading WebmasterWorld, a forum for website publishers and SEO experts to speculate about the Google algorithm, discuss AdSense tricks, and debug website design issues.

Lots of studying and reading

After I got to Stanford, I took lots of great computer science classes like CS106X and CS107, and I also started section leading the CS106 classes.

I spent almost all my time outside of class reading about design, programming, browsers, and JavaScript. Like 4-5 hours a day.

What did I read? Lots of different stuff. But, mostly blogs by first-class designers and programmers who I admire a lot. Real badasses. For a sampling of some of these blogs, take a look at the “Respect Rollcall” in the sidebar of this blog.

A viral hit

Then, in the summer of 2010, while interning at Facebook, I built YouTube Instant to settle a bet with a friend. It’s a video site that lets you search YouTube in real-time. The site went on to get 1 million visitors within 10 days of launch, and the YouTube CEO also tweeted me a job offer. Read about the media frenzy here.

I know that YouTube Instant’s success was mostly due to chance good timing and a little luck. Read None of Us Knows What We’re Doing for some more of my thoughts about that.

The beat goes on

I noticed that lots of people were using YouTube Instant to listen to music videos, and that got me thinking about other cool ways to use the YouTube API. So, my friend Jake Becker and I decided to spend the first 3 months of 2011 building Instant.fm, a really easy way to share music playlists with your friends. We both learned a ton of new stuff during this project.

Some things we mastered during this project:

• jQuery
• CSS (and Modernizr and YepNope for cross-browser issues)
• Python
• Tornado (web framework/server)
• Git (version control) & GitHub
• Last.fm API
• YouTube API
• Working on a team

And some other things we learned how to use, too:

• Nginx (web server)
• Supervisor (to daemonize Tornado)
• SQLAlchemy (ORM)
• Apache Ant (to build and deploy the site after a push)

Read more about all the tech we learned in Instant.fm Tech Stack.

TL;DR – Just start building stuff!

The point of this long expose on everything I’ve built since age 11 is that if you want to learn programming, then you need to start building stuff! Right now. No more excuses.

Doing something is the fastest way to learn it.

Reading a programming language book from front to back is boring and you’ll quit before you finish it. But, if you have a project in mind, you can learn what you need to know as you go along, which is more effective both in terms of speed and mastery of the content.

Computer science classes

Taking CS courses at a university is another great way to learn programming. Most good CS curriculums emphasize learning the important concepts and paradigms in the field of CS, as opposed to teaching a specific programming language. This can be an eye-opening experience for self-taught programmers who’ve never had any formal education.

I remember sitting in my first-ever CS class at Stanford (a class taught in C++) thinking “How on EARTH can they have variables that don’t start with dollar signs?” Up until that point, I’d only ever programmed in PHP! It took a while for me to drop the habit of $putting $dollar $signs before every variable name!

Work at a software company

Another way to get much better at programming is to work at a software company like Facebook or Quora, which I did over the last two summers. You’ll learn how to program from people way better than you, how to read and understand other people’s code, and how to work on large projects with a team.

Still — more than anything else — the very best way to learn programming is to do side projects. Have I repeated this enough times, yet?

How to learn programming:

• Do side projects.
• Buy and read programming books.
• Do side projects.
• Take computer science classes.
• Do side projects.
• Read programming blogs.
• Do side projects.

That’s the best advice I got.

Happy hacking!

This was originally posted as an answer to “How did Feross Aboukhadijeh learn to program?” on Quora.

Follow me on GitHub!

I’ve been using Github for the past year to work on projects with friends (like Instant.fm and iBoard). But until now, I’ve only created private repositories.

I’ve been meaning to open source the rest of my projects on Github, but haven’t got around to it, until now!!

Right now, I’ve mostly posted some small projects, but I plan to use Github for pretty much anything I build, from now on.

Something I’ve needed to do from time to time is spoof my computer’s MAC address. This is useful for debugging network issues or temporarily getting onto the Stanford Wi-Fi network when my physical MAC address changes, such as when Apple replaced my logic board (motherboard).

For the uninitiated, a Media Access Control address (MAC address) is “a unique identifier assigned to network interfaces for communications on the physical network segment”. You can read more on Wikipedia.

Despite the fact that all network devices (laptops, iPhones, routers, etc.) have physical MAC addresses burned into the hardware, you can actually change (or spoof) your MAC address completely in software, and other network devices will only see your spoofed address.

Other Solutions Sucked

I was disappointed with the Mac OS X offering in this area. None of the existing stuff worked well. The biggest annoyance with most of the solutions I found was that the Wi-Fi card (Airport) needs to be manually disassociated from any connected networks, in order for the change to be applied correctly. Doing this manually every time is annoying.

My Solution

So, I made a Python script that lets you change your MAC address in one command. Check it out and download it from Github. Improvements welcome!

I tested it on Lion 10.7, but should work on 10.6 and 10.5 with slight modifications.

Usage

Note: Use en0 for wired ethernet and en1 for wireless. From the terminal, run:

sudo python SpoofMAC.py INTERFACE MAC_ADDRESS

Example:

sudo python SpoofMAC.py en1 12:12:12:12:12:12

That’s it! Let me know in the comments if you find this useful.

I’m humbled by the way YouTube Instant took off so quickly after I built it and I’m extremely thankful to the hundreds of thousands of good people who tweeted, liked, stumbled and otherwise shared my little hack and helped turn it into a worldwide sensation for a few weeks.

Thanks to the good people at .Net Magazine for putting on this awesome award series to recognize so many of the creative people who work in web design and development. Here are the nomination categories:

#3 Best API Use: YouTube Instant
#12 Brilliant Newcomer Award: Feross Aboukhadijeh

Visit the .Net Awards site to vote & view all the nominees

Related posts:

1. YouTube Instant Shortlisted for Best API Use in .Net Magazine Awards

The Myth of the Superhuman

They believe, it seems, that “successful people” like Steve Jobs, Mark Zuckerberg, Bill Gates, Paul Graham, Walt Disney, etc. have some sort of in-built superhuman awesomeness that makes them smarter, cleverer, more brilliant than the rest of us.

They believe that something about these people is unique, that their feats would be unachievable by anyone else.

To be fair, it sort of makes sense, right? I mean, these folks succeeded spectacularly where thousands of others failed forgettably. So, the successful people must be more brilliant than average folk like you and me, no?

An Anecdote: YouTube Instant

Something about this line of thinking doesn’t sit right with me. I know from my own experience building YouTube Instant that success oftentimes just happens without any foresight, pre-planning, or exceptional skill.

When I built YouTube Instant, I was certain that someone else had built something like this before. Sure, I thought it was a neat idea, but it didn’t seem that special to me.

I certainly had no idea that writing those 190 lines of JavaScript would change my life forever.

I never could have predicted that such a simple hack would go on to get 1 million visitors in 10 days, garner worldwide media attention, and earn me a personal job offer from Chad Hurley (who was YouTube CEO at the time; now the owner of Delicious). How could I possibly have predicted all that?

The truth is, I nearly watched a movie with my roommate instead of building YouTube Instant that night (the movie we were considering was Grave of the Fireflies if you’re curious). Needless to say: I’m so happy that I decided to code instead of watch a movie that night.

(Note: I am not comparing YouTube Instant to Apple, Facebook, or Microsoft. Building my little tech demo was nowhere near as awesome as starting a company. Yet, I still think that there are interesting lessons to learn from YouTube Instant that apply equally to the founders of Apple, Facebook, and the rest.)

So, what lessons does this teach us?

Lesson #1. None of us knows what we’re doing.

I wasn’t being especially brilliant when I thought of YouTube Instant. I didn’t have an epiphany, a secret how-to guide, or any awareness about the viral potential of the site. It was simply a random idea, like any other.

Here is the truth. None of us knows what we’re doing. We are all just winging it. Yep, that’s right. Even Fortune 500 CEOs, Nobel Prize winners, and U.S. presidents — all are really good at winging it.

The sooner you realize that no one knows what they’re doing, the sooner you’ll lose your fear of uncertainty and just go for it. Successful people aren’t by necessity any smarter than the rest of us.

Lesson #2. Don’t listen to successful entrepreneurs.

The folks who succeed have no way to know if their success was due to talent, skill, and planning, or merely dumb luck.

If you ask them though, they’ll confidently spout reason after reason why they — and no one else – could possibly own 90% of the desktop PC market, or whatever they achieved. In their minds, it couldn’t have turned out any other way. Most of this is just after-the-fact rationalization, though. The truth is, they succeeded and have no idea why. They’re just explaining it in the best way they can.

(I also think that some successful entrepreneurs are self-conscious about their success. They are afraid that people might think their success was only luck. So they start company after company trying to replicate their first success. On the other hand, most entrepreneurs just love building stuff — it’s in their blood. It’s hard to tell the two apart.)

Lesson #3. You can’t predict what will succeed.

With YouTube Instant, almost everything was luck. I was lucky to hear about Google Instant right when it came out. I was lucky to have the idea for YouTube Instant pop into my head — it was completely random. I was lucky to have 3 free hours that day to build the site (yep, I built the whole thing in 3 hours). I was lucky that someone I’d never met before decided to share the link on Hacker News. I was lucky that Chad Hurley was reading Hacker News when the link was on the homepage. And so on…

The only thing that was in my control, however, was whether or not I decided to show up.

Ninety percent of life is just showing up. — Woody Allen

I decided to build YouTube Instant instead of watching a movie. That decision was the only thing 100% in my control. I decided to turn off the incessant trivial chatter on Twitter and TechCrunch, get my hands dirty, and just build something.

Several Hacker News commenters lamented the fact that they didn’t build this first. Others derided the idea as obvious and trivial, and therefore not worth building. (post 1, post 2)

This comment captures my thoughts on the matter perfectly:

[YouTube Instant is] using existing mechanisms in YouTube’s JSON API to find suggestions and play them. All the true “magic” is provided by YouTube’s engineers. Not to take it away from someone who writes the right lines of JavaScript at the right time, but you don’t have to place the guy on a pedestal.

The real lesson here is for yourself and others here who think like you — if you tell yourself that things like this take superhuman efforts, how will you ever be able to produce something creative yourself?

Exactly.

Lesson #4. You can create your own luck.

Building stuff that people use — even stuff that goes viral — doesn’t take superhuman effort. It just takes a willingness to get your hands dirty (as discussed above) – something that most hackers and entrepreneurs already possess.

If you keep building stuff, you create your own luck. That’s how you learn and get better. But, first you have to believe that it’s possible.

Best thing to do: always believe that extraordinarily epic wins are possible. Conversely, don’t worry if you fail — just move on to the next thing.

Recently, a researcher studied the lives of 400 people over the course of 10 years and watched for any lucky breaks or chance encounters — both good and bad. Here is what he found:

My research revealed that lucky people generate good fortune via four basic principles. They are skilled at creating and noticing chance opportunities, make lucky decisions by listening to their intuition, create self-fulfilling prophesies via positive expectations, and adopt a resilient attitude that transforms bad luck into good.

Lesson #5. Don’t compare yourself to others.

Look at the code for YouTube Instant. It’s butt ugly. I was pretty awful at JavaScript 9 months ago.

If I had compared myself to Brendan Eich, John Resig, Paul Irish, Alex Sexton, or other JavaScript badasses I would have just got depressed and never built anything. Better to write ugly code with a bunch of hacks than to write nothing at all.

When I interned at Facebook last summer, I heard from other engineers that early Facebook code quality was pretty awful. I also heard that Mark Zuckerberg wasn’t exactly the best programmer in the world (not sure how accurate this is, but I heard it from a few people).

The point is: he did the important things right. Zuck had great product vision and he didn’t fret about writing perfect code.

Done is better than perfect.

Bonus: Lesson #6. Stop reading Hacker News.

Hacker News is awesome — don’t get me wrong. It’s a great way to follow the goings-on of the hacker community. But, like everything else in life, it’s subject to the law of diminishing returns.

Easy question. Which is better: 3 more hours reading Hacker News? Or, 3 hours spent building YouTube Instant?

Neo, sooner or later you’re going to realize just as I did that there’s a difference between knowing the path and walking the path. — Morpheus, “The Matrix”

Thank you for reading. Happy hacking.

First, what features did we want to build?

Playlist Creation

• No login required.
• Build your playlist on site, or upload a .m3u, .txt, or .pls file from iTunes, Windows Media Player, or WinAmp.
• Each playlist gets a unique, shareable, short URL.
• Allow background image uploading. (TODO)

Playlist Editing

• Drag and drop to re-order songs.
• Buttons to Move To Top, and Kill Song.
• Change playlist name/title with an inline edit (no refresh).

Playlist Viewing

• Use YouTube as audio source.
• Shuffle, repeat, show/hide video.
• Suggest songs to be added to playlist. (TODO)
• Keyboard shortcuts for power users.

Social Features

• Share playlist on Facebook, Twitter.
• See what your friends are listening to. (TODO)

Mini-browser

• Pane which behaves like an iOS navigation view, with a stack of ‘views’.
• Allows searching and browsing artist/album information without stopping the music.
• Clicking links doesn’t cause user’s browser to leave the page.
• Uses fancy animations, which look great.

Non-stop Music

• On most music sites, clicking something stops the music (very jarring).
• On Instant.fm, everything is AJAX, so nothing stops the music.
• Even logging in/out works without a page refresh (the correct playlist edit tools are shown/hidden).

Browser Support

• All modern browsers.
• Internet Explorer 8. (TODO)

How We Built It

HTML5 Boilerplate

• Rock-solid default for HTML5 websites.
• Build script for minifying and hyper-optimizing JS, CSS, and HTML.
• CSS reset, base styles, cross-browser normalization, non-semantic helper classes.
• Server side optimization to reduce total page weight.

CSS3

• 1430 lines of hand-written CSS.
• CSS3 hotness (transitions, box-shadow, border-radius, gradient, box-reflect, text-shadow).
• Degrades gracefully in older browsers (…most of the time).

JavaScript Libraries

jQuery 1.6

• The best.

jQuery UI 1.8

• The only feature we used was the excellent “sortable” module.

Modernizr

• HTML5 feature detection in JS and CSS.
• HTML5 shiv so semantic elements like <header> and <footer> work in IE.

YepNope

• Conditional JS resource loader for polyfills.
• Asynchronous script loading.

Backend Stuff

Tornado Web Server

• Asynchronous non-blocking Python web server.
• Modules (tornado.database, tornado.httpserver, tornado.web, tornado.auth, tornado.ioloop).

Nginx web server

• We run 4 Tornado processes, and use nginx to load balance between them.

Supervisor

• Process control system used to daemonize the Tornado server instances.

SQL Alchemy

• We were doing raw SQL for a while, then we got tired of that and decided to go with an ORM.

Python

• Server-side image resizing.
• Playlist file (.m3u, .txt., .pls) parsing after upload.

Apache Ant

• Custom build script to deploy new code.
• Currently, the site goes offline for about 30 seconds whenever we deploy (we think this is acceptable).

Web APIs

Last.fm API

• Artist, song, album information (summaries + pictures).
• Powers our search results.
• Uses HTML5 Local Storage to cache search results.

Facebook API

• Facebook Connect.
• Social plugins (Like button, Comments widget).

YouTube API

• Search API.
• Embedded player JavaScript API.

jQuery Plugins

jQuery Templates

• Render data into a template and insert into DOM.
• Officially supported plugin.

JSizes

• Adds support for querying and setting additional CSS properties.
• Min-width, min-height, max-width, max-height, border-*-width, margin, padding.

Jeditable

• Adds Edit-in-place functionality to forms (no page refresh).

Auto Expanding Text Area

• A la Facebook and Quora.

ColorBox

• Light-weight, customizable lightbox.

Graphics

Helveticons

• High quality commercial icon package.

Photoshop

• All other images, icons, logo, custom missing album/artist images are ours.

Misc Notes

InstantFM.com

• .com TLD redirects to our main .fm site, in case users get confused with the .fm extension.
• Purchased from a domain squatting company (BuyDomains).

Git

• Commit early, commit often.
• 686 commits so far.
• We used GitHub.

Conclusion

As you can see, we built Instant.fm upon a solid foundation of awesome. Most of our tools are free software with great documentation and easily hackable code. Jake and I used a bunch of different tech on this project, since we wanted to try out some new things. Plus, playing with new frameworks and libraries is always fun.

We learned a ton during this project. I hope that this information has helped you. If you found this useful, leave a note in the comments.

Related posts:

1. Eric Ries’s Lean Startup – Tech Talk

Instant.fm (built in 3 months)

I was planning a more elaborate blog post to announce this site, but it’s been live for a while now and it’s not a secret anymore (was it ever a secret?). Instant.fm is the culmination of a 3 month long collaboration with Jake Becker, a good friend (and fellow Stanford CS student).

We both realized after YouTube Instant went viral (1 million visitors in 10 days!) that it was very handy for discovering music videos, and that’s the basis for this new site.

Here’s an interview we did with All Things Digital. Read the full article here.

The main idea is to make it easy to publish and share music playlists with your friends using YouTube music videos as the source of the audio.

Instant.fm allows you to upload a music playlist (exported from a music player like iTunes, or created from scratch from songs on the site). When you create a playlist, you get a unique URL to share it on Facebook and Twitter so your friends and followers can check it out.

We know that there are lots of sites out there that let you share playlists, but none make it as simple and intuitive as we think it should be.

Now, since we launched the site we’ve learned that many of our core assumptions about how people share music weren’t 100% correct, so we’re planning a revamped Instant.fm that will be pretty different. I’ll write another blog post in the future about what we’ve learned while building Instant.fm — there’s quite a lot we’ve taken away from this project.

Instant.IO (built in 8 hours)

Sheesh, I think I’m definitely obsessed with the word instant. I admit, the name “Instant.IO” doesn’t tell you much about this site’s purpose, but I had the domain name and I didn’t want to let the $100 registration go to waste (yeah non-standard TLDs are pretty expensive).

Anyway, you have to try Instant.IO for yourself. Trust me, it’s worth doing once (and maybe only once). I built the site in 8 hours during the Stanford Hack Competition which I helped to organize. This was the first hack competition that my club, the Stanford ACM, organized.

We had Adam D’Angelo (founder of Quora), Andrew Bosworth (creator of Facebook News Feed), Steve Bourne (creator of sh), and Steven Schlansker (founder of Trumpet Technologies) judge the hacks. See the results.

The whole event went splendidly — we’re planning to do this every quarter from now on. Fun times.

Instant.IO re-uses the Google text-to-speech API scraper I built when I made Brain Grinder last month. I’m planning to open source the code for this (and more) sometime soon.

Update Sep 1: Source code for both Instant.io and BrainGrinder is now available on Github.

Emu Spin (built in 30 minutes)

Yeah, what can I say about this site? It was an inside joke that (sort of) unintentionally went viral.

If you’re curious, I built the site as a joke for Jake (my Instant.fm partner). Jake has long used this emu picture as his Facebook profile picture. And for some inexplicable reason the emu got rotated 90 degrees at some point (either a bug in Facebook or he accidentally rotated it). After that, it became a running joke and it’s been rotated several times since then.

I rotated the emu once when Jake stepped away from his computer, and I can only assume that others have done the same. EmuSpin.com is a tribute to Jake’s spinning emu.

Emu … Spin?

I didn’t think anyone but Jake would enjoy the site, since it’s an inside joke. I suppose I should have known better. An odd-looking animal spinning around for no reason = instant interweb lulz. Get the source on Github.

EmuSpin.com made it to Reddit. (should’ve guessed it). Here are a few of my favorite comments:

I was emused.

Surprisingly difficult to turn away from.

Anyone else expecting something profane to pop up at any moment?

Stare at his beak for a long time then look away

I can’t get drunk and write a coherent paragraph; who does LSD and builds an entire website?

A rival appears against nyan.cat!

BEST WEBSITE EVER.

I love the Internet.

Your Turn

Thanks for reading! Have you built anything cool recently? I’d love to check it out. Leave a comment below.

UPDATE: 100th Blog Post!

I just realized this was my 100th blog post! Woo hoo! It’s been too much fun. My sincere thanks go out to all 251,003 visitors for reading 402,946 of my blog posts since I started this thing in 2009.

Here’s to another 100 posts!