diff options
Diffstat (limited to 'FUTURE')
| -rw-r--r-- | FUTURE | 335 |
1 files changed, 335 insertions, 0 deletions
@@ -0,0 +1,335 @@ +SWIG-1.3.12, SWIG 2.0, and Beyond +================================= + +With the release of SWIG-1.3.12, I thought I'd take a few moments of +everyone's time to talk about the past, the present, and the future of +SWIG development. I'm really quite excited about the current release +because I think it represents a huge turning point in SWIG's +development. Furthermore, it is only the beginning of bigger and +better things to come. However, we definitely need your help. + +To put a little perspective on the discussion, I'd start with a few +development statistics. In the last 12 months, there have been over +300 entries added to the CHANGES log and over 4000 CVS commits. +Although that may not sound like a lot compared to a huge software +project, it is significant in the context of SWIG. As a point of +comparison, there has been more SWIG development this year than in any +other year of the project and more than in the previous three years +combined. This even includes the first few years of development in +which there was also a lot of activity. Furthermore, many of the +recent changes have been extremely non-trivial (e.g., templates, +namespaces, type system, operators, etc.). As a result, SWIG is more +capable than I ever imagined possible. + +Regrettably, I must admit that I've been a little negligent in +discussing the roadmap for where I thought this flurry of SWIG +development was actually headed. In part, this is because I've been +buried in work. However, the real reason is that I didn't really know +where we were going---except that in a time far far far away in the +future, we might arrive at some kind of "stable" release with a +version number other than "1.3.x". Needless to say, that's not a very +compelling story. + +That said, I've spent a lot of time thinking about SWIG and trying to +wrap my brain around it. Specifically, just what is (or should be) +the primary focus of this project and what are we really trying to do? +That's what the rest of this message is about. + +SWIG Prehistory +--------------- +The first version of SWIG was written in 1995. The original system +was developed to help with some software wrapping problems I +encountered while writing molecular dynamics software at Los +Alamos. Later that year, I became interested in extending the wrapper +generator to support other scripting languages so it was rewritten in +C++ and modified with multiple backends (Tcl, Perl, and Guile). This +led to a first public release in February, 1996. Feedback from this +release led to a series of enhancements and the release of SWIG 1.0 in +September 1996. Throughout this process, my intent was to create a +tool that I would want to use---I never viewed the project as an +CS experiment in programming languages or software engineering. + +SWIG 1.1 +-------- +SWIG-1.1 (June, 1997) represented a series of enhancements that were +added in response to feedback at conferences and from users. Shadow +classes, exception handling, typemaps, and a number of more useful +features were added. However, the overall structure of the system was +relatively unchanged from the initial version. Following the release +of 1.1, a series of minor patch releases were made. This resulted in +the release of SWIG-1.1p5 in February, 1998. Unfortunately, this +release would remain the last "stable" release for quite a long +time---in fact, it is still listed as the last "stable" release on the +SWIG web page! + +SWIG Hell +--------- +Even during the development of SWIG-1.1, it was clear that the whole +design of the system was deeply flawed. The implementation was a mess +and the C/C++ support was full of holes and nasty corner cases. +Furthermore, there was no unifying principle that tied all of the +different SWIG directives together. Not only that, fixing these +problems appeared to be nothing short of impossible---requiring a +total ground-up rewrite at best. The only redeeming quality was that +the system basically worked "well enough," it was extensively +documented, and its flaws mostly known. People could use it and there +were work-arounds for most common problems. + +To deal with the design problem, there were at least four attempts to +completely rewrite SWIG, some of which were attempted in parallel with +the work on SWIG-1.1. Unfortunately, none of these were ever +completed. The primary problem was a strong "second system" effect and +a desire to make SWIG do everything that one might conceivably want to +do with a wrapper generator (somehow). Clearly, this was a recipe for +disaster. In fact, all such attempts to rewrite SWIG were eventually +abandoned several years ago. In hindsight, I think the real problem +was that these rewrite efforts focused far too much attention on +implementation technique rather than principles. In short, the +failure of these efforts was due to a lack of clarity in understanding +how SWIG ought to work (regardless of how it was actually +implemented). + +SWIG Restart (1.3a1-1.3a5) +-------------------------- +Having languished for several years, the SWIG1.1p5 release had a +growing pile of maintenance issues. It didn't work for newer versions +of certain language modules and a lot of minor bug reports and feature +requests had been building up. With a lot of help from Loic Dachary and +Thien-Thi Nguyen, we put together the 1.3a1 release (February, +2000). This was mostly a bug fix release to 1.1p5 except that the +preprocessor module from SWIG1.2 was added and a lot of minor +enhancements were added. + +For the next six months, a massive effort was made to rewrite all of +SWIG's internal data structures (strings, lists, hashes, etc.). This +work was all going on underneath the covers while we tried to keep +SWIG in an operational state. The primary focus of this work was +really one of cleanup. Having given up on a total rewrite, perhaps +we could settle with making the implementation incrementally better +than before. In addition this, Matthias Koppe jumped on board to +reimplement the Guile module and to make other improvements to the system. + +An important aspect of these releases was that many parts of the +system not directly related to wrapper code generation were removed. +This included the documentation system and Objective-C support. These +were not removed because they weren't useful. Instead, the +documentation system was removed because it accounted for nearly half +of the special SWIG directives, yet had no direct bearing on what SWIG +actually did. Obective-C support was removed because it was tangled +with C++ support in a way that was very difficult to understand and +maintain. The removal of these features was seen as a way to vastly +simplify cleanup--and to buy some time where we could rethink their +role in a future release. + +SWIG Redux (1.3.6-1.3.11) +------------------------- +This work, started in February 2001, is the beginning of the current +SWIG implementation. With the help of William Fulton, Matthias Koppe, +Lyle Johnson, Luigi Ballabio, Jason Stewart, Richard Palmer, Sam +Liddicot, and others, this work can best be described as the wholesale +destruction of everything remaining from SWIG-1.1. The language +module API, type system, the parser, numerous SWIG directives, and +SWIG library files---all destroyed or rewritten. Not only that, we +started introducing significant incompatibilities with +SWIG-1.1---mostly in an effort to correct past wrongs and get +ourselves out of the tangled mess of earlier versions. A huge number +of long-standing bugs and difficult feature requests have also been +resolved. + +The general goal of this development could best be described as an +attempt to reduce SWIG to an easily described set of general "ideas" +about how it should operate. Special SWIG directives have been +eliminated or combined with others. Different parts of the system have +been better integrated. Features not directly related to wrapper code +generation have been removed and the system has become more +focused. Changes in internal data structures and APIs have allowed +SWIG to capture more information from interface files and to resolve +hard corner cases. More often than not, these are things that you +never notice unless you are an old user and you suddenly realize that +a problem you had several years back has disappeared. + +Along with the destruction of old code, this work has quietly +introduced a new core--the most significant features of which are a +new C++ type system and multi-pass compilation. More importantly, +this work has really tried to provide a more principled foundation for +future SWIG development. However, just what is this "more principled +foundation?" + +Convergence (1.3.12) +-------------------- +With the upcoming 1.3.12 release, SWIG is about to take a big leap +forward. Almost all of this is due to one realization---that almost +every hard problem in past SWIG releases has been directly related to +errors and limitations in its type system. Types are the key to +understanding the structure of C and C++ code. They are at the heart +of understanding advanced language features like namespaces, nested +classes, and templates. They are directly related to the data +marshalling that occurs in wrappers. Not only that, they interact +with nearly every SWIG directive. A proper type system *is* the +necessary foundation for moving SWIG forward. + +To be honest, I don't think that the emphasis on types is entirely +obvious. In fact, a great deal of work in past SWIG rewrites has +focused on the problem of C++ parsing. For instance, modifying the +parser to handle more advanced C++ code or representing parse trees as +XML. Furthermore, if one looks at the SWIG mailing list, you can find +a *lot* of messages related to issues of C++ parsing whereas almost no +one ever talks about types (well, other than typemaps). Even other +wrapper generation tools seems to spend a lot of time dealing with the +parsing issue. Although parsing is clearly important, I don't think it +has ever been the real problem in SWIG. This is because even though a +parser can tell you what's in a header file, it doesn't tell you +anything about how the different pieces of the system behave or how +they might interact. To do that, you need to do a lot more than just +parsing--and that's really the whole point. + +Although earlier 1.3 releases have made big improvements to the type +system, SWIG-1.3.12 is the first release that really tries to tackle +the type-system issue in a major way. We have patched nearly all +remaining holes in the type system and we have added full support for +C++ namespaces. Not only that, we have completely reimplemented C++ +template support in a way that supports templates, member templates, +and template specialization. Luigi and I are currently using this to +work on proper SWIG library support for parts of the C++ standard +library and the Standard Template Library (STL). Although some crusty +C programmers (present company excepted), might balk at such apparent +insanity, this work has impacted all parts of SWIG at all levels. +Even a variety of subtle errors in C support have been fixed by this +work. + +In addition to the type system work, SWIG-1.3.12 contains continued +reduction in the implementation. Directives have been removed, +refined, renamed, or consolidated. We're still narrowing the focus of +the system and working towards some kind of convergence. "Convergence +to what?!?", you ask. + +So, what is SWIG? +----------------- +In a nutshell, SWIG is a C/C++ declaration compiler that generates +wrapper code (okay, so you knew that much). However, to really +understand what SWIG is doing and where SWIG-1.3.x is headed, it is +useful to know that the whole system is essentially being built around +three extensions to the C++ type system: + + - Typemaps. Typemaps are rules that define the process by which + data is converted between languages. They are fully integrated + with the C++ type system and they are applied using a type-based + pattern matching mechanism. All data conversion SWIG is + defined by typemaps and is fully reconfigurable. + + - Declaration annotation. There are special directives that modify + the wrapping behavior of individual declarations. Declarations + can be selectively identified and decorated with arbitrary + attributes that affect wrapper generation. Like typemaps, + declaration matching is fully integrated with the C++ type system. + Almost all SWIG customization directives are a form of declaration + annotation. + + - Class extension. The ability to extend classes and structures + with new methods/attributes when building wrappers. Classes + are part of the type system so class extension is naturally + integrated with the C++ type system as well (big surprise). + +And that's it--this is the end-game of SWIG-1.3.x development. When +stabilized and documented it will become SWIG-2.0. + +The Bigger Picture +------------------ +I really want to emphasize that all of this work is part of a much +bigger picture. SWIG is used by a surprising number of people in +industry, government, and academia. It's used to build systems used +by millions of people every day. It has even shown up in video games +and other unlikely settings. Although SWIG doesn't have the same +visibility as many large software projects, over 12000 people have +downloaded SWIG-1.3.11 in the last 4 months. Clearly someone is using +it for something! Because of this, I think it is important for us to +work on moving SWIG towards a more solid foundation. Doing so will +give the system more credibility and long term viability---and it will +be a lot more fun to use! + +It's also worth noting that there are some rather interesting CS +connections at work here. Extensions to the type system and typemaps +have some interesting relations to work in programming languages. The +SWIG declaration annotation system and class extension feature seem +oddly similar to work in the emerging area of Aspect Oriented +Programming (AOP). There are undoubtedly connections to other areas +of software engineering and architecture. + +The key point is that SWIG isn't going to connect to anything if +no-one can quite describe what it is or how it works. + +SWIG-2.0 and the Future +----------------------- +SWIG-1.3.12 still represents work in progress. There are bugs, the +documentation is still incomplete, and there are parts of the +implementation that are rather ugly. We are also still working out a +few very hard problems like nested classes, callback functions, and +overloading. A few old features are still missing (Objective-C, +documentation). However, I believe the end of the 1.3.x series is +near and achievable. + +Over the summer, a few more 1.3.x releases may appear but the current +plan is to aim for a SWIG-2.0 release in September. This release is +really moving towards the design principles described above and will +be a very major enhancement over SWIG-1.1. + +As for the future, a number of interesting ideas are on the table. I +want to add support for contracts/assertions in order to solve some +reliability issues that arise when retrofitting legacy codes with a +scripting interface. Support for an extension language has been +promoted by David Fletcher and was suggested by someone else on the +mailing list rather recently. I have a graduate student working on +SWIG support for the Microsoft CLR and .NET languages. Other work +might include support for alternative parsers, dynamically loadable +language modules, and so forth. However, none of this is really going +to materialize if we can't get the 2.0 release stablized. In fact, I +see SWIG-2.0 as a necessary step for moving forward with these ideas. + +We need your help! Yes, you. +---------------------------- +Nobody gets paid to work on SWIG. The developers are volunteers who +work in their spare time. Furthermore, SWIG is not supported by +investors, a large corporation, or research grants. I work on it +because it's fun, challenging, and useful. I presume that other +developers do the same. However, we only have limited resources and +we need your help. + +- If you like SWIG and find it useful, we need you to try new versions. + We want you to torture test the releases and to break them. We need + bug reports. No bug is too obscure or unimportant---we *will* fix it + if we can. We also need feedback about things that are annoying or + compatibility features that might help in going from 1.1 to 2.0. + +- We need help with documentation, examples, testing, libraries, and all + sorts of other aspects of the release. Even if you have never + written a SWIG language module or dived into its implementation, + there are many ways that you can help. Consider writing a case study + about how you wrapped a big library. Contribute tests that break the + implementation in horrible ways. Help with the web page or FAQ. + +- Most of the SWIG-1.3.x work has focused on the SWIG core. However, as + the 2.0 release nears, we will be working on a variety of enhancements + to the language modules. If there are things you would like to see + in any of the language modules, you need to let us know. + +- There are SWIG language modules that have not made it into the + distribution. Examples that I know about include ITCL, Swig-Eiffel, + and Swig-Lua. We would gladly make these part of the standard SWIG + distribution. However, we also need help to do it. Porting from + SWIG-1.1 is no easy task, but we're more than willing to help. It's + not as bad as one might imagine. + +- We are always looking for developers. Subscribe to + the swig-dev mailing list, details at http://www.swig.org/mail.html, + or send me email to get involved. + +Acknowledgements +---------------- +I'd just like to thank everyone who has submitted feedback, bugs, made +contributions, and put up with my occasional thrashing over the years. +I welcome any comments about this document and how we can make SWIG even +better. + +Dave Beazley (beazley@cs.uchicago.edu) +June 2, 2002 + |