I am a web developer, born in PERU and now living in PDX. My platform of choice is CFML but I'm always open to learn anything. I currently work for Mentor Graphics.

I also do freelance web development and have a few open source projects that I maintain.

Where To Find Me:

A Rant on Frameworks, Libraries, and Shells

May 30, 2008

We spend ridiculous amount of time discussing the merits (or lack of) of the different MVC frameworks in ColdFusion, and we keep doing it again and again. Well, at least everyone should agree that when it comes time to decide how to go from one page to another we got that pretty much well covered. But for all the OO fever taking over the hard-core ColdFusion community as of lately, why are we still focusing on the presentation layer still, which honestly is the least OO place in the whole application? Where are the repositories of just plain libraries? components and subsystems that we plug together to provide sophisticated functionality to our applications regardless of how the presentation layer is done.

Yes, we have Transfer, ColdSpring, and Reactor, and they really rock; but it seems that that's where the list ends. Has nobody written a really tight, super strong, caching library? what about a job-scheduling library? It could be that the lack of these libraries as separate entities is what motivates MVC framework authors to add more and more complexity to their projects, which in turn fuels the discussions of why these frameworks are so bloated and complicated. I know Luis, from ColdBox, has put a lot of effort in building the caching functionality in his framework, and knowing the quality of his code I'm pretty sure it kicks a$$; but sadly is all tied into the framework. Wouldn't it be much more beneficial to everyone if there were already some project out there that would provide this functionality so that you could just plug it in into your application?

If we have enough of these little libraries that would focus on doing one thing and one thing only, then the 'framework' in which an application is built would not need to be these gigantic-swiss-army-knife platforms that we are getting used to. They would become lighter and lighter until they are reduced to a 'core', just down to the basic principle that embodies their core ideas. Imagine a coldbox-core, a modelglue-core, a fusebox-core, super small code bases that just provide the basic guidelines and minimal code to support a proposal on how an application should be structured; And then on top of them we could have 'distributions' or 'shells', canned projects that would group together a 'core' and a few libraries to put together a basic solution, custom tailored for specific purposes. For example, you could have a shell for an intranet application that includes the framework core, a caching subsytem, a security subsystem, an IOC subsystem and an ORM subsystem; or on the other hand there could be a shell for a plain website that may only include the framework core and a caching subsystem. The bottom line is that all those parts can be plugged in an out and replaced or extended, and the application as a whole could be customized to the specific needs and would not need to carry any added complexity that is not desired by the author.

If I may give a suggestion, we should move on from the MVC frameworks and start focusing in these kind of projects. We need to focus more on libraries and subsystems in that the only interface is an API that can be used by other applications. Of course, in order to do that, it should be easy for everyone to find these libraries. RIAForge makes a terrific job as a repository of entire projects, and maybe it could be extended to add a new category exclusively for ColdFusion libraries and APIs; or maybe we need an entirely new website to allow us to post and search libraries for different types of functionalities that anyone wanted to add to their application.

Any thoughts?


Posted on May 30, 2008      18 Comments

Ben Nadel's Gravatar Ben Nadel ...
I think the problem with this is that it really requires the "Best in Show" of every type of sub-system. I think that's the only want that true plug-n-play will happen. Otherwise, you are gonna get something that only works, maybe 50% of the time, and when it doesn't work people have to roll their own.

For example, let's say I have an App where the only "caching" system I need is this:

Application[ objectKey ] = object

Done. I just cached a singleton. But, on the other hand, you get something like Transfer, where the caching needs to have soft references and objects that cache based on "need algorithms" and suddenly, that is a whole different level of caching.

So, ok, maybe these are two different caching strategies that require two different plug-n-play modules. But what happens when someone else has a different need .... and a different need.... and a different need. Suddenly, you have all these versions of caching mechanism. And then where are we - right back to where we started, with everyone having their own copy of a caching mechanism.

The problem with one-size-fits-all desires is that we do NOT have one-size-fits all needs. And to roll solutions that are one-size-fits all, they probably have to be 99% bloated to handle the flexibility.

Think about security. How can you have one app that has NO users, just a sense of logged in /logged out and a different app that has users, groups, permission levels, and active ActiveDiretory integration both using the same module?? You simply can't - not in a clean way at least.

I don't think you want go this route. Or maybe I am not understanding what you mean??

Look at production of cars. Each car is a fine-tuned machine. Different engines, different chasis. Most things are different. Does that make them worse? No, it makes them "specific". It makes them taylored.

... just my thoughts.
May 30, 2008 10:27 AM
Jim Priest's Gravatar Jim Priest ...
@Ben - Most cars are NOT unique machines. Most cars share common, standardized engines and chassis across product lines. They may be slightly tweaked for different performance objectives, etc - but the basic parts are the same. If auto makers had to make unique parts for each vehicle they would quickly go out of business!

I'm working on a simple Mach-II CVS export today - I think this is a perfect example of the thing Oscar is talking about. Why should I have to hack this up from scratch? I agree it's difficult to come up with a 'one size fits all' solution - but I think some basic modules could be created.
May 30, 2008 10:47 AM
Ben Nadel's Gravatar Ben Nadel ...
@Jim,

Yes, cars do share parts, but these are the small parts. Its the configuration of these parts that are unique. I think this can be ported over to ColdFusion as well... we all share the same core functionality - tag set and built-in functions. We may even have basic UDFs that we like to share. But, to me, that seems like the most of what is hugely plug-in-play. Once we need to wire those all together into meaningful files, I think it just gets too complicated.

Do you mean CSV? I am not familiar with CVS....

But, let's assume you do mean "CSV" - comma separated values. At first, you think that you can easily wrap this up and package it, right? I mean, that's a pretty small piece of functionality. But think about it. What is it? a UDF? Sure, it can be, I have made several for CSV. But where does it live? Do you have to create and package it as a CSV component? Or, do you use it in part of your own UDF library? If it is a component, I am sure it has lots of additional stuff for going from queries to CSV, arrays to CSV, and may stuff for the other way - going from a CSV to a query. Or maybe converting CSV to an Excel.

This all sounds really awesome. Until you look up and realize that all you needed was a query to CSV function and now yo uhave a 4,000 line component that does 10 different functions. So now what? Do I have to go in a strip out all the junk that I don't use? Do I have to find the one function that I need, and copy it into my UDF library?

Or... do you just cache that component and use the 1/10 of it that you need. I mean, maybe the other 9/10s will come in handy later on.

Of course, this problem exists in all walks of life. I have a TV that does a 1000 features, and I only watch 4 channels. Clearly I am not going to strip out what I don't need :) I'm gonna pay the $400 and just deal with it :)

So, if that works in the real world, then maybe that's cool in the programming world as well.

I hope I don't come off sounding condescending - I am quite literally thinking as I type this. I am so used to hand coding almost every single aspect of my projects that I have trouble even thinking about taking a black-box of functionality and applying it.

What I *have* seen though, with my own eyes, is people take black-boxed functionality and apply it horribly. I fear that this will take an already lazy programmer and make them lazy & sloppy.

... anyway, just some thoughts. I have never dealt with creating packaged modules, so really this whole talk is beyond my expertise.
May 30, 2008 11:03 AM
Terry Schmitt's Gravatar Terry Schmitt ...
I've also been reading all the posts lately about frameworks. One of the most interesting is regarding the OpenBD Admin project where the framework developers and other strong advocates of frameworks are opting NOT to use a name-brand framework.

More inline with your post, I too, have been thinking about a plugin architecture for frameworks. Have a core framework and then install only the plugins that you require. I think the plugins would need to be framework specific, as writing agnostic plugins that could be utilized by all frameworks would be next to impossible.

From a performance perspective, I sometimes look at a Fusebox parsed file and cringe. I look at the steady stream of new features in Coldbox and wonder if we aren't paying a performance penalty for a feature-rich product, especially when we are only using 20% of the features for a given project. This is not a ding on the framework developers. They are all way smarter then myself and my hat is tipped to them!

Most of my work is all behind firewalls with a realatively light user base, so I gladly use frameworks for 99% of my programming and give up the n% of overhead from the framework. Out in the real world it may be a different story, though, although hardware is pretty cheap these days and you can always throw more horsepower at an application. My holy grail is still to have the most efficient code that I can.
May 30, 2008 11:15 AM
Jim Priest's Gravatar Jim Priest ...
@Ben - I did mean CSV :) Working on a CVS project as well and that's just too confusing!

This is uber-complicated but what would be cool is if your 'central library' had a wizard. So I pick "CSV Component" and it offers me up the available functions. I select what I want and what framework I'm using and it builds my component which I can download and plugin to my app.
May 30, 2008 11:34 AM
Ben Nadel's Gravatar Ben Nadel ...
@Jim,

Now that would be totally badass :)
May 30, 2008 11:42 AM
Oscar Arevalo's Gravatar Oscar Arevalo ...
@Ben,
I think I may have not explained well myself. I am not advocating for someone to build the "mother of all frameworks", something that would be a unique solution for every single thing. Actually I think thats the way our current frameworks are headed and something we have to avoid at all costs. What I'm saying is to focus in creating and promoting more subsystems, just plain libraries, things that may be composed of a single CFC or 300 CFCs and helper files, but that are completely encapsulated. Yes, its a black box and you use it at your own risk and only if it is worth to you. But nobody said that there has to be only one ORM or only one IoC container, tomorrow someone may want to build another one that could maybe have less features but be more efficient, or a lot bigger and more "enterprisey". The more libraries we have at our disposal, the more options that we as developers can have for picking the right components for our applications.
May 30, 2008 12:56 PM
Sana's Gravatar Sana ...
@ Oscar
I do agree 50% but there are many cases where users ask for features should be integrated with MVC.
Also most of the ColdBox plugins are developed by others and we are still considering offer more features by additional small utilities(plguins).

@Terry ColdBox does not load any extra feature if you don't activate them. Huge list of features are asked by coldbox users and to be honest not sure how to avoid this situation. as you know every body like ready made solution by just couple lines of code and job done.



@
May 30, 2008 01:02 PM
TomdeMan's Gravatar TomdeMan ...
Patience, Patience, Patience...

The majority of CF developers are evolving into 'real' software engineers and architects.

The popularity of MVC frameworks has brought light to the core concepts and fundamentals of building software, such as Design Patterns. If anything, working in the different frameworks and seeing the differences between them is an easy way to relate programming concepts to actual CF code. Which for most, is a huge time saver, since going back to school and holding a full time job these days is a bit much.

Which brings me to my next point...Time.

With all these rapid development languages popping up, the more tools that are provided for us without having to google, browse thru some tutorials or docs, and tinker with configuration, the more time we save. It would be great to have an al la carte system where you could add and remove libraries with a click. Who's gonna build it? I have to say if I see anyone coming out with it it be Luis and it be an additional section to the CB Dashboard, I believe he has even talked about it before, but I don't think its in his road map.

The ideal solution would be to have some kind of a standard between the frameworks for extending to such a system and integrating external libraries. Maybe we need a Frameworks Standards Commission - hehe.

Now that CF developers have risen to a new level its only a matter of time before we take the next step. I think you have hit it right on the nose. The MVC framework debate has been beaten to a pulp.

I am with you on moving on to libraries and subsystems. However, if I had to squeeze something else in before hand, it would be DOCUMENTATION and EDUCATION. If we are going to provide more then we need to keep learning and understanding what is already available.

Coldspring and Transfer are perfect examples, and both come from programmers with a strong Java background and experience with true OO programming. Some have unlocked the full power of these, others use what they need never caring to dig in.

The closer we get to understanding and implementing true programming fundamentals and software development practices the faster we will evolve.
May 30, 2008 02:37 PM
Luis Majano's Gravatar Luis Majano ...
Great post Oscar, finally some funkyness!!

We talked about this a long time ago, where ColdBox could be assembled at download, and I am still considering it. The plugin architecture and interceptor architecture in ColdBox is exactly what I envisioned in increasing the framework's functionality without touching the core. The coldbox core is very small and fast, the rest of the features come thanks to the plugin/interceptor architecture. The thing is that, you have to download all the plugins/interceptors as of now and then decide if you want ses, utilities, security, etc. They all get brought in to the life cycle as you request them.

The only difference is that you have to clean it up manually as of now if you don't want those files on your server. My debate with that is, what harm does a file do if its there? It is not being used, so who cares. Makes my work easier, since then I don't have to build a "Customize your ColdBox download" wizard (Unless somebody would want to do it??)

Anyways, I agree with a lean-mean core and that is what ColdBox at least has become after several iterations. It does MVC real well, now if you want more functionality, well ColdBox is also a toolkit (library) and you can bring in anything you want to your application, and share your plugins/interceptors and be merry and happy. Most of the time when you use a platform or library, you don't use ALL of the features and functionalities, but that doesn't mean that they get loaded or used. So in my eyes, I don't worry about this problem, because I know I can build tons of libraries of plugins/interceptors to enhance my applications. One feature that could sneak in the Dashboard is a community repository of plugins/interceptors. So when you start a new app, you can pick and choose which plugins/interceptors you want to download and use in your application (Maybe Tom's new security interceptor or Ben's XML parser). All without touching the core!! Now that would be funky!!
May 30, 2008 03:11 PM
Oscar Arevalo's Gravatar Oscar Arevalo ...
@Luis,

That sounds very cool, I may have to take a new look to ColdBox again. The plugins and interceptor libraries are something similar to what I'm saying, except that those plugins -as far as I know- are specific to ColdBox. I am talking about "framework-agnostic" libraries. I don't know how much this applies to ColdBox in particular or if it is even possible, but thats a whole different discussion thread.
May 30, 2008 05:38 PM
Mark Ireland's Gravatar Mark Ireland ...
Wouldnt a ''Business Logic" module be the place to start?

Something better than a set of switches?
May 30, 2008 08:21 PM
Mike Brunt's Gravatar Mike Brunt ...
I spend most of my time in "the engine room", the JVM solving performance issues. I currently have 4 clients with falling over memory leak problems, everyone of them is using a combination of mach-ii or ModelGlue, ColdSpring and Transfer. This may be a pure co-incidence and I cannot say, as yet, that these frameworks are part of the issue. However it does concern me somewhat that very few in our community are paying enough attention to the effects on performance from complex and abstracted mechanisms which as a result of the complexity are very, very hard to troubleshoot for end users.
May 31, 2008 07:23 PM
Mark Mandel's Gravatar Mark Mandel ...
@Mike - I actually agree with you a lot.

Having worked with you on occasion, and looking at the gruelling stress tests you put applications through, it opened my eyes to the sort of pressures these frameworks can come under.

Much to your credit, Transfer has several stress tests that I put it under now to track memory management, and see how much performance wise it can handle.
May 31, 2008 11:08 PM
Mike Brunt's Gravatar Mike Brunt ...
Mark thanks for your comment I very much appreciate it. Recently I was presenting at WebManiacs in Washington DC on performance tuning etc. There were a good number of participants, around 40 I think and I asked how many had participated in any load testing of any kind and 3 raised their hands. Sadly it is unusual for any applications to be load tested before reaching Production and that is at the root of why there are many web applications out there in some kind of trouble.
June 01, 2008 11:19 AM
Peter J. Farrell's Gravatar Peter J. Farrell ...
I find it funny that nobody mentioned that both the logging and caching packages that were introduced in Mach-II 1.6 are framework agnostic. Of course we've written loaders so we can use them at the controller level in the framework, but it was direction that both of these packages can be used without the need of using Mach-II's MVC framework. I.e. independent packages

For example, all you have to do for the caching package is instantiate a CacheStrategyManager (which parent/child aware), load in a caching strategy (two are currently available in SVN or write your own) and use the API provided to put, get, remove, flush or keyExists. The two strategies currently offered are a time span cache or LRU cache. An EhCache strategy to talk to an Eh is on the way. We even have a CacheStats.cfc so all caching strategies can report stats in a uniformed manner.

The same applies for the logging package. We've written a loader so Mach-II can use the package internally, but it's trivial to instantiate the LogFactory, add a logger and use the API. We currently have three loggers in the repo at the moment -- CFLogLogger, EmailLogger and a Mach-II specific MachIILogger (for Mach-II debug output at the end of a request).

Both packages use their own Abstract.cfc so it's trivial to write your own caching strategy or loggers. In the case of the logging package, you could write a logger that uses a bundled logging adapter (like the scope adapter) and the bundled channel filter.

All in all, we've made sure that both the packages are framework agnostic and can fully be used independently of the MVC framework aspect of Mach-II.
June 03, 2008 01:23 AM
Oscar Arevalo's Gravatar Oscar Arevalo ...
@Peter,
That is awesome, I didn't knew that about Mach-II; but that was exactly what I was trying to say, to have the framework core to focus on only one thing and rely on external independent packages for additional features like caching and logging.

I will definitely take a look at these features. Are these packages available by themselves?
June 03, 2008 02:40 PM
Peter J. Farrell's Gravatar Peter J. Farrell ...
@Oscar,

The packages themselves are not available yet independently from the core download since we're just about to enter a beta period. Given enough interest is wouldn't be hard to create individual zips. However, both packages live in their own top level directory so you all you have to do copy those folders out currently. Definitely something to think about offering them as separate packages. Would you be willing to open a ticket for us about this at trac.mach-ii.com?
June 03, 2008 03:39 PM

All comments are moderated to prevent spam
Leave a comment