Biggest lessons I learned from developing open source libraries

I've been developing open source libraries for more than 10 years. During the years, there are painful lessons I've learned. In this article I will summary the lessons, so you can learn from the lesson and don't repeat the mistakes in your development.

Most lessons are learned from my C++ cpgf library.

Lesson 1 -- be too ambitious and develop too large libraries

I started cpgf library from very small ideas. I noticed there is no std::function in C++03 and it's so useful, then I developed Callback. And I'm a big fan of event dispatching pattern, then I developed CallbackList. That's the original cpgf.

Then I thought it's not cool that C++ lacks the C#/Java style reflection system, and reflection is useful. Also the callback related code are great to support the reflection system, I need some way to use the callback. For these reasons, I added the reflection and meta data system. To support the reflection system, I had to add another component, the variant, which is complicated and ugly.

The idea of the reflection system is good (it's still good now). We only need to construct the meta data once, and the universal meta data can be used everywhere, such as property editor, script binding.
Oh, script binding, it's charming. I was a veteran in gaming development, and I knew how valuable script binding in C++ is. And script binding is a good camp to test the reflection system. Then I added Lua script binding.
Lua is too simple to bind, I wouldn't stop there, so I added Python binding, then JavaScript V8 binding, then JavaScript SpiderMonkey binding.

What's next? It's time to script binding some real world C++ code. Then I added meta data for STL, Box2D, OpenGL, SFML, and Irrlicht. During the working, I found it's really boring to create the meta data by hand, then I developed a meta data generating tool written in Java.

While all above components are related, I added another component, the tweening library, which has nothing to do with any above component. I wanted a tweening library, then I added it.

You can imagine how much work needed to develop the library. I spent all my part time in at least three or more years on the library.

Now let's summary the components in cpgf:

• Callback and callback list
• Variant
• Reflection
• Lua script binding
• Python script binding
• V8 script binding
• SpiderMonkey script binding
• Meta data for various libraries
• Meta data generating tool
• Tweening library.

Seeing the components, seems I was a super man, while I'm not. The library is usable, and has decent quality, but it has a fatal problem -- it's too too too large. If each component is split to independent libraries, each library is non-trivial and some may be still quite large (see script binding).
If the library was developed by a large software company like Microsoft, most likely it needs 20 to 100 programmers on the library, not including any non-programmer staff such as product manager. While I'm only the main person working on the library... (there were some persons contributed to the library, I would say thanks to them).

The problems in the too large library:

• Even if I work on it full time for next 10 years, I still can't finish all features nor I can make it good enough. The library scale is far beyond a one-person project.
• Difficult for users to use. If a user only needs the callback system, he has to build the whole library, and 99% code is not needed.
• It's too rigid. Even though I designed the components in very loose coupled way, changing one corner may still affect the while library.
• The development process is difficult. If I want to fix a bug in the callback, the whole library will be compiled, and if I don't have Python development toolkit installed, I have to disable the Python binding to avoid compile errors.

Lesson 2 -- be too conservative on new technology

At the time I started the cpgf project, the latest C++ standard was C++03. C++11 was already drafted, and was partially usable in the compilers. Especially, TR1 was there already.
I chose to stick to C++03, because I wanted my libraries be able to be used in as many as compilers.
That's another big mistake. The different between C++11 and pre-C++11 is nothing like C++03 and pre-03, or between C++14 and C++11. C++11 is a new era of C++.

Without C++11, I couldn't use variadic template, so I had to use template partial specialization to simulate variadic template. The technology is spread in the whole callback and reflection components. It not only causes very ugly code, but also makes the code much less readable and hard to maintain.
Beside variadic template, I also couldn't use shared_ptr and unique_ptr in TR1, so I had to reinvent the smart pointers in cpgf. Big pain, seriously wasting time!

Now my mind still prefers to that a library should be compatible with as many compilers and platforms as possible, but compatibility is not the only factor. If the newer standard can ease my work significantly, I will compromise it.

Lesson 3 -- be too closed and not using modern open source infrastructures

At the beginning time of the project, Github was just emerging, not popular yet. And at the time I was still using SVN for version control. But there were quite some VCS for open source projects that allows collaboration between developers.
However, what I did is to host the SVN on a private repository, and provide the source code package download on my website. I wanted to open source the project, but I used a very closed source way. Apparently that will prevent any other developers from joining the development.
Fortunately, after several years, I moved the project to Github.

Don't repeat the same mistake

Despite of cpgf is successful or not, I'm still a big fine of event dispatching pattern (observer pattern), so I decided to develop another library eventpp.
In eventpp, I avoided all mistakes I made in cpgf by:

• I only focused on event dispatch related stuff, I don't add any irrelevant features to the library. That will keep the library small, concise, and concentrated. I've ever been keen for adding Python style multi-threading task queue to eventpp because it's easy to implement using EventQueue, but I suppressed the impulses because it's not related to event dispatching.
• I write the library using C++11. Without C++11 I won't start the project. C++11 is enough and working very well for the library. I avoid any later C++ standard such as C++17, because some development environment, especially the embedded industry, may be still using a compile quite behind the newer standard. And the later standard doesn't gain me much benefit comparing to C++11.
• I put the source code on Github since the first day. If there's no Github, I will put it on Gitlab, if there is no Gitlab, I will put it on what ever public repository, instead of any private repository.

The result is good. After two years since the first release, I'm still happy to work on eventpp. And eventpp has much higher quality than cpgf, and it gets more stars and forks than cpgf, and more users.