Writing a CKEditor Plugin

Exposition: Why should you care about a random text editor?

Modding CKEditor is a well-designed exercise in self-flagellation.

CKEditor is a text editor plugin that works with rich text (bold, italics, tables, links, etc.). If you’ve ever used Gmail or Hotmail, you’ve used something similar.  CKEditor is relatively easy to use, powerful and has lots of features, and it’s used everywhere. But while the code is well written and well documented, at a macro level there are terrible design things going on, and if you want to modify it, you’ll want to shoot yourself shortly thereafter.

Since I’ve been wrestling with it for quite a while now, I wanted to share my thoughts:

• In case you happen to be forced to alter CKEditor as well. Hopefully this will be a useful breadcrumb trail.
• To convince myself I’m not faking it. CKEditor has well written source, so someone could argue “maybe they’re doing it well and you just don’t get it.”
  This is perfectly valid. The best way to figure out if I’m faking it is to actually try and explain why it’s bad, and see if this post turns into “CKEditor is an architectural marvel” in the process.
• Complaining is whining unless you can explain why, and how to make things better.
• I need to understand this stuff myself, and nothing helps with understanding like trying to explain.
• Because I’ve been staring at this for days and need to say something.

Act One: Parallel Universe APIs

Half of CKEditor code essentially replicates JavaScript DOM APIs and jQuery. Let’s use the most radical example of CKEditor manipulation – the dialog architecture.

CKEditor dialogs are used for special properties. If you want to add a link, you would create a dialog to set the URL.

What language do you think dialogs would be designed in? Presumably a language that is used to lay things out in a web browser. Hmmm…what language do we use for that…HT nope, you’re right, it’s JavaScript:

The new dialog system in CKEditor is to be written from scratch. One of the key things in CKEditor is that it doesn’t rely on HTML pages to run. Everything is created on the fly in JavaScript. In this way, we avoid limitations about cross domain serving of the editor code, like CDN solutions, and enhance the editor performance.

http://docs.cksource.com/FCKeditor_3.x/Design_and_Architecture/Dialog_System#Dialog_Definition

We’ll get back to the benefits CKEditor claims later, but let’s think about this. You’re laying out a mini-web page. You’re a web developer and probably lay out web pages all the time. You read A List Apart obsessively; you understand the difference between content (HTML), presentation (CSS), and behavior (JavaScript), and the benefits of separating them.

And then you see this:

(This is actual dialog code that CKEditor uses to show a link dialog.)

Is that HTML? Yup – it looks like a <select> – there’s a list of options (items), and an id, and even a label.
Is that JavaScript? Yup – it’s got an onChange, a setup, and a commit, whatever that is.

This confuzzlement is compounded by the fact that there are about 1000 (literally, see for yourself) lines of this JavaScript/HTML hybrid. If you separated this into HTML and JavaScript, it would be easy to read, not indented six ten tabs in, and it wouldn’t take you an hour of sorting to figure out what a dialog that’s basically a handful of selects and some text fields is doing.

By the way, there’s CSS too:

		var basicCss =
			'background:url(' + CKEDITOR.getUrl( this.path + 'images/anchor.gif' ) + ') no-repeat ' + side + ' center;' +
			'border:1px dotted #00f;';

Let’s establish some of the downsides of this mixed-content architecture:

• Wading a thousand four-hundred line morass of mixed behavior and content every time you want to change things
• No way to test the layout of your dialog without loading all of CKEditor (as opposed to HTML, which can be tested by anyone with a web browser)
• You have to relearn all your HTML/JavaScript/CSS ninja skills in a weird JS hybrid
• Oh and if you miss a brace, it’s not like missing an angle bracket – everything dies. JS is brittle. Did I mention you have to reload all of CKEditor to test your fixes?

So how about the benefits? The CKEditor docs claim that you can do cross domain* loading (for content delivery networks, etc.). While there are plenty of things wrong with the solution, the CDN problem is serious and it would be irresponsible to ignore it. Is there another way we can allow CDN loading without replacing all of our HTML with JavaScript?

One option: Write HTML for the dialogs, but convert it into escaped JavaScript internally. Escaped HTML is ugly, but if we compile it, we needn’t use it except in production.

This rant is just barely getting started, but I’ll stop here for the moment.

From a code perspective, it’s easier to add new features than remove old ones.

From a user perspective, it’s much, much easier to add new features than remove old ones.

Nobody knows about your new feature, but there are at least three people who love that button you added to the toolbar two years ago. Even if no one else cares, they’ll make sure to post angry requests on your forums asking why you got rid of it. If you’re not careful, they’ll accuse your company of “not caring about its users,” or they might even say you “used to be great, but ever since they got successful they’ve started to go downhill”…ever hear that before? [1]

At the same time, it’s important to cut features, or you end up sliding down this curve:

So how do you decide what to cut?

Why Not Just Count?

One of the great things about analytics is getting real data about who’s using what. Every month, Google Analytics sends me an email that tells me exactly what people are reading on this blog. (Hint: It’s not the minesweeper articles)

That gave me an idea: Why not generate similar analytics for a library like jQuery?

There’s a nice list of sites using jQuery available here:

http://docs.jquery.com/Sites_Using_jQuery

Let’s take a site and see what it’s actually using:

(Speaking of wanting to stick to old things, Match.com uses jQuery 1.2…)

Match.com’s landing page is simple enough that we can do an actual list of what calls they make to jQuery, which I’ve uploaded here.

From this, we can tell that the standard selector function $() gets used a lot, as does readCookie and addClass. From there, the next step is to figure out what they’re actually calling.

In the case of jQuery, this is not too hard in principle. Get a version of jQuery, add a line that increments a counter every time a function is used, and inject it into your favorite website. It’s a little too involved for this post, but perhaps in the future (earlier if someone decides to jump in and do it!)

[1] Back in the 90s, Microsoft obsessed about making sure everything from the past worked, even if it mean keeping broken things from earlier versions of Windows:

The Windows testing team is huge and one of their most important responsibilities is guaranteeing that everyone can safely upgrade their operating system, no matter what applications they have installed, and those applications will continue to run, even if those applications do bad things or use undocumented functions or rely on buggy behavior that happens to be buggy in Windows n but is no longer buggy in Windows n+1. In fact if you poke around in the AppCompatibility section of your registry you’ll see a whole list of applications that Windows treats specially, emulating various old bugs and quirky behaviors so they’ll continue to work.

The problem: Why is this d_mn thing taking so long?

Ever since I started working on my own, I’ve spend far more time figuring out how to motivate myself. Because I’m working in a startup, I make sure to do just the minimum amount of work to make a viable product, so that I can save time and work on other things.

And yet, I find myself staring into space, going on Google Reader, Hacker News, playing games, or anything I can find aside from work.

As I’m talking with a friend online, we make some observations:

• While the eventual result is interesting, the work I’m doing is boring.
• The reason I’m bored is because the work is easy.
• Therefore, if I want to be more productive, I need to make my work harder.

I throw out the disorderly, simple, get it done code that everyone says you should write for a prototype. I make things organized. I put all my variables into neat, sensible structures that follow design principles. I use a regular expression to replace 6 lines that I could’ve done by hand in five minutes.

All of a sudden, I’m getting work done again.

What gives?

–

I’ve talked with scores of startups over the years, and they consistently err on the side of overbuilding rather than underbuilding. Fancy prototypes take months to code, months that are wasted if your idea of a customer is the two people you told at the startup meetup you went to last week that said it looked kind of cool; people create entire frameworks to build sites that could be done in the time it takes to write a shell script (that installs WordPress).

Thus, the rule to build fast and build cheap was born.

Am I some kind of unlikely exception? Perhaps. But after giving it more thought, I realized that the need to make some things hard is not surprising, and not unique.

–

Are you working at a startup?

Why?

The pay is (sometimes) lower and the responsibilities are (often) greater. Most of the programmers I know working on their own projects are exceptionally good at programming. The hackers aren’t hanging out at the back of the bell curve. They can pass FizzBuzz, they know it, and that’s exactly why they’re finding a place where they live or die by their skills.

So it’s not entirely surprising that, if you pick people who are fairly competent at coding, so competent that they want to go to a place where they may be solely responsible for the technical success of the product they build, that they might just prefer not to always have simple, somnambulist-friendly code assignments.

Now, as with any essay that argues a point, you can poke holes in this thesis. Perhaps, you might say, the original rule isn’t against giving yourself a technical challenge from time to time, but rather as a safety railing to stop the people who want to make Pandora Craigslist Farmville with an extra layer on top so you can tag your singing cows with pictures from your last missed connection. And that’s a fair point.

What we do thrives on exceptions. I wouldn’t be able to think of a good example, but by chance, I read of one in the NYT yesterday:

When asked what market research went into the iPad, Mr. Jobs replied: “None. It’s not the consumers’ job to know what they want.”

Do what works.

For anyone familiar with dbgsrv, it is similar in purpose and function to msvsmon, the visual studio remote debugging monitor. Both applications are “process servers” for their respective debugging environment. Running dbgsrv on a remote machine allows developers to attach to any process on that machine, or launch processes under the debugger.

Using dbgsrv is slightly mysterious, as the UI for using it is a bit cryptic in WinDbg. I’m going to run through a few examples to show how useful dbgsrv can be.

To start, we need to get dbgsrv running on the machine you want to debug. Usually, this will be a test computer that you want to debug from your development computer. If you want to follow along the examples, you can use a single machine, although this scenario isn’t very useful for real world debugging. To start dbgsrv, run it from the command line giving it parameters to pick a transport to use. Usually, tcp is the easiest when not working with domain-joined machines:

dbgsrv -t tcp:port=31337

If there are no errors, dbgsrv will start silently and start listening for connections. (You may see a firewall dialog at this point). To use the process server from windbg, use the “Connect to Remote Stub…” menu item from the File menu:

The connection string should match what you’ve started dbgsrv with. For instance, if dbgsrv is running on my computer named “timtst”, I would connect to “tcp:port=31337,server=timtst”. After accepting the connection string in this dialog, there will be no indication of failure or success until you actually try to use it. Once the connection string is entered, you can try to attach to a process using File->Attach to Process, or using F6. If the connection was successful, you will see a list of processes on your test computer, or an error if the connection was not successful.

To use the same functionality in cdb/ntsd, you can use the “premote” command line argument. For instance, to use the same dbgsrv instance as in the previous example to attach to process ID 4000, you can use the following command with ntsd:

ntsd -premote tcp:port=31337,server=timtst -p 4000

Using ntsd also lets you start a process under the debugger through a process server:

ntsd -premote tcp:port=31337,server=timtst C:\MyTestApp.exe

Note that the path given refers to a path on the remote machine running dbgsrv, not the machine running ntsd. This functionality is not available in windbg.

There is an excellent write-up with more information about dbgsrv at nynaeve.net

If you’ve found this interesting, I highly recommend the book Advanced Windows Debugging
by Daniel Pravat. As one of the most informative books about the Windows Debugging toolset, it has a host of information about dbgsrv and the rest of the tools.

1. Problem 1: Beautiful Soup isn’t supported anymore
   Beautiful Soup is the de facto HTML parser. Beloved by Python programmers, it’s capable of dealing with broken and messy HTML. Sadly, the libraries that it used are being replaced, and the main developer doesn’t have time to work on it anymore.
   Solution: This was the easiest problem to deal with. I asked the New York Python Meetup, and they all recommended lxml.
2. Problem 2: lxml doesn’t have a Python 2.7 build
   The easy solution – “easy_install lxml” – is supposed to get an egg file precompiled with lxml’s dependencies (at least, says the INSTALL file in the download).
   There were two problems:
   1. It doesn’t
   2. None of the .exes on the site are for Python 2.7.
   Solution: As it turns out, there’s a way around this dilemma; someone’s posted a script to build it online. It’s only for 32-bit though, it seems, but I gave it a short spin anyway.
3. Problem 3: Cython error
   Solution: This fix was easy.
4. Problem 4: “vcvarsall.bat” missing
   As it turns out, building many Python packages requires vcvarsall.bat, which is probably a compiling tool of some kind in Microsoft’s toolchain. The fix that comes up in search engine results involve hacking in a different compiler (gcc from MinGW), which I suspected might cause other incompatibilities.
   Solution: After talking with a friend from Microsoft, I determined that downloading and installing the Windows SDK would be a good place to start. Though that didn’t work, I did end up installing Visual Studio C++ Express, which did include vcvarsall.bat.
5. Problem 5: “vcvarsall.bat” missing
   For some unknown reason, even after adding vcvarsall to the path,  the error still came up.
   Solution: It was at this point that I realized that the build script was for 32-bit. If I was going to go through the trouble of trying it, maybe it would be worth trying a 32-bit precompiled exe, which I ended up discovering on the same site I visited earlier.
6. Problem 6: Installation didn’t work
   Even though the install went find and “import lxml” worked without a hitch, the lxml package was strangely empty – there was nothing in it!
   Solution: I went through site-packages and cleaned it out – there were two separate lxmls in there from my previous experiments. Removing one of them cleared it all up.

(I recently had an experience where I couldn’t beta test some software because it was built for 32-bit, but my computer was 64-bit. As it turned out, that didn’t apply here.)

Possible lesson: If something is unlikely to work, but is easy and quick, try it anyway.

Programming languages are great because they make it easy to write code – in a way that makes it easier for us to understand.

In the end though, all of these abstractions were not written with your problem in mind, but a generic, off-the-shelf problem that you don’t actually have.

Thus, write with any language you want. But write in a way that’s appropriate to your problem, and write in a way that’s appropriate to you.

My version: Build every one to throw away.

Fred Brooks’ classic The Mythical Man-Month made this observation developing a new piece of software – inevitably, many of the problems that will need to be solved won’t be known until you try to solve them. The first try will incorporate many feelings of should-have and would-have that will be fixed on the second try.

This is much like a journeyman craftsman building his or her first bench, or planting their first garden. The first try is a learning experience. There is no way around this, nor should there be – learning is often best by doing. The valuable asset is the experience of the builder, not the product.

But we don’t live in the Middle Ages, the age of guilds. We live in the post-Industrial Age. Factory owners don’t consider hoarding all the toys or cars or electronics they produce as an asset. Henry Ford’s genius was to redesign not just the golden eggs, but the machine that lays them.

As programmers, we are not working on a code base. We are working on a machine that produces code. The machine is made up of us, our experiences, the tools we build to make code (which can be made out of code themselves!). This is where results come from, and we should spend our time tuning this machine by producing more, not protecting what we have.

Does this mean that we should throw out old code? By no means! Old code is one of the most efficient resources we have for producing new code. But every process of manufacture in the past has been made into a more automatic and refined process. How could we consider ourselves any different?

Does this mean that we should code like crap? By no means! As was once said, “If you write the first one to throw out, you will end up throwing out the second one as well.” The point is, write good code, but be willing to write new code – the point is to make not good code, but a good code factory.

[more (full article)]

I was just as much shocked as dismayed. Mulberry was hawked by so many colleges that I assumed its destiny was all but sealed. College students, as early adopters, would all be exposed to Mulberry and see how wonderful it was, and they’d take it to their jobs, promoting an almost viral spread.

I guess the Thunderbird/Outlook duopoly was just too strong for Cyrusoft to handle. However, all is not lost, as Mulberry is available for free now.

Let’s take a look at all the wonderful features of this program! (full article)

[originally started a long time ago]

Security is not just about locking down a system from a list of attacks. The way you design a UI dictates how people behave, and people’s behavior is responsible for 90% of attacks. [citation needed] Defaults matter. No one forgets to lock an automatically locking door.

The fact is, while a security system must be set up to prevent hacking attacks, guesswork, and theft, it must also be designed in such a way that leads people to behave more securely. When you have a system where keys are hard to create but easy to copy, naturally, people will end up using the same keys at eBay that they do at Flickr.

The need to “educate users” is an indication of design failure.

How could I justify something so outlandish? Simple. The difficulty of a change (how many hiccups, how many bugs) depends not so much on its scope as it does on how well planned it is. Inevitably, we underestimate how much planning small changes take, and overestimate when we see big, scary changes. Does this mean rewrite is easy? Of course not. But it’s not impossible, or as costly as we think, and just the opposite is true for minor changes.

*The usual caveats: If you have a problem that can be solved with a small change, I recommend sticking to that before resorting to bigger changes. But once you start seeing so-called small changes multiply, you may have hit the stall point.