Open Source & Strategy
Wow, what a great time at CascadiaPHP! I want to personally thank everyone from the community who managed to come out to help make the conference double as our first concrete5-con!
Slack is great. We're there if you want to join us, just goto:
The newly-forming official concrete5 security team is currently looking for new active members from the community to join our team.
If you know a community member who has experience in security and would be available and suitable for the position, please nominate him or her - you can also nominate yourself. If you are nominating someone else, please make sure that you've asked their permission.
Time to vote for your favorite talks at the OpenWest Conference!
Voting for talks at the OpenWest conference is open until March 31st, 2015.
OpenWest is a community-driven conference that focuses on all things Open Source, Open Hardware and Open Standards.
Presentations are chosen using blind community voting as the main factor in deciding which speakers to invite.
We are proud to announce that the new version 7 beginner and developer online training classes on concrete5.org went live today.
I’m proud to announce Jess has got Version 7 Level I - Editor certification out the door and live! Woot woot!
We started many moons ago with a little marketplace of just our own add-ons. It was exciting to have found a revenue model for our free software.
As we looked for ways to differentiate ourselves from the sea of other CMSs, the idea of a clean, supported marketplace like Apples App Store made a lot of sense. We started accepting submissions from 3rd parties. With the excitement of Christmas morning, Andy and Ryan would personally review each submission as we had time, of which there was plenty of to go around.
Then things grew, and grew, and grew - and we couldnt keep up. So we added the PRB as an open source inspired committee of reviewers who could collaboratively help alleviate some of the initial poking around at submissions, letting us just be the final approvers. For a while, it worked great. We knew which add-ons to look at first, and while everyone wanted things to move faster, the marketplace maintained a level of excellence far above our peers.
As the marketplace grew, there were struggles around serving both the developer and site owner side of the equation in one environment. (e.g. What impact should difficulty level have? Whats the best way to install themes on existing vs. new site?) Theres a disconnect between our original vision and the reality today. We hear from our 3rd party developers that they feel like theyre putting great effort into providing support, and we hear from customers that its not always delivering the experience they thought they were getting. For example, just this week I was told:
"...In my opinion you guys should rise bar with quality control. This is second time in three years that I'm seriously thinking walking away from concrete for exactly same reason. Poor quality of the apps and bad support. I think you should rate developers for support so before we by their item we know who we deal with..."
Its natural for something important that is growing to have adolescent pains as things change. We believe there are two major problems with the PRB and marketplace today:
- We are trying to serve a diverse customer base with widely different expectations for what a finished product and good support means with a single experience. Is our marketplace a code repository for concrete5 developers who want to share with each other and maybe make a buck? Or is it more like Shopifys app store, with add-ons that are all safe, tested and running against a consistent environment in the will that always works no matter your technical ability? While we still believe we can serve everyone, we need to make some changes to speak more directly to the extreme ends of the spectrum.
- Due to demands on our time, we are unable to keep up with approval of new work in anything close to a timely fashion.
Its time for some well considered changes to improve the processes of the PRB and give different types of marketplace customers the experience they need. Heres whats happening
Submission to the PRB and getting a listing live quickly
1) The automatic checking system (called linter) has been improved. It now runs both on new submissions and new versions of existing listings.
2) If your new submission passes all the tests, you are given a launch date of 1 week in the future. Your listing will go live in 1-week even if no one on the PRB approves it. It will have a Has not been approved by PRB member! message on the listing, you'll have to flip a check box to get them to show up in the marketplace, and it will not be shown in concrete5s in-dashboard shopping UI.
3) The PRB will now have a small administrator group. They can approve add-ons and themes without anyone from the core team signing off. This adds their personal endorsement to a live listing, and/or launches the listing right then with their endorsement.
4) If your new submission did NOT pass all the tests, your listing will not go live until one of the PRB administrators or a core team member launches it.
Submissions will be rejected with limited cause or explanation. (e.g. wouldnt install, code looked sloppy in first file I opened spend some time working on this and resubmit plz. ) The PRB is not a free school, thats for the forums.
5) A new public Submitting to the marketplace forum area will be created for the entire community to help people get their code right.
6) Issues with version updates that fail linter tests will be worked out in that public forum, and not in the existing PRB interface.
1) PRB administrators and regular members have term lengths of 6 months. At the end of that period they automatically lose their PRB membership and have to request it again.
2) It takes 3 PRB administrators agreeing to accept a new PRB member.
3) It takes unanimous PRB administrators and the core team agreeing to accept a new PRB administrator.
4) To start we appoint: JohnTheFish, Mnkras, GoutNet, Tallacman as the first PRB administrators. For this special situation, their terms will be staggered (9, 8, 7, and 6 months respectively. ) Everyone else has had there term end and should re-apply now if they're interested in being part of the new PRB.
5) To request to join as a PRB member, submit your concrete5 user name to this google spreadsheet where PRB admins will review submissions on a monthly basis: (http://bit.ly/1vSIRZ3)
1) We will have a linter server you can just throw stuff at and get results from sans PRB. Dont abuse it or it will go away.
2) If the community wants to build additional checks for the linter to run, weve open sourced the framework and an example test. The PRB administrator group will decide if your check will be added.
3) The PRB interface has changed to be more usable. Were collapsing information that was getting in the way and were adding some filters/status to help keep things organized.
It All Started with Files
In concrete5 5.2.0, we introduced a completely new file manager. Files were now full-fledged objects with extensible metadata, permissions and sets. Retrieving information about a particular file object was much more expensive (in terms of database queries) than it used to be.
All of a sudden, certain add-ons like galleries which gathered large amounts of files at once were quite slow. They had to run hundreds of queries just to render a page.
Caching to the Rescue
In web applications, caching typically means retrieving an object from a faster place be it RAM, or a serialized object in a hard disk than reconstituting it entirely from the database. This lessens the load on the database and can dramatically improve performance. We added our own custom caching layer, which stored objects in the file system, and this problem went away. We liked this solution so much that we added more objects to this cache.
Enter Zend Cache
When we started working on localizing our add-ons (so they could be included in translation) we realized that it would make a lot of sense to use Zend Translate, which is a library contained with theZend Framework. Zend Translate makes extensive use of Zend Cache. We took a look at Zend Cache, and realized that without a lot of work, we could remove our custom caching layer, replace it with Zend Cache, and automatically get access to different cache backends, like APC, Xcache, memcached and more. It was a no-brainer.
Not entirely. Zend Cache hasalways had its share of quirks. The biggest one? The default cache, which stores objects in the filesystem, gets slower and slower the more objects you place in it, due to some garbage collection routines that Zend Cache uses. If you disable garbage collection, the cache is fast, but it fills up with millions of files. This was exacerbated by the fact that, as we added features like extensible attributes, layouts and full page caching, we found ourselves storing more and more objects in that cache.
And the worst part? While we were trying to lessen database access, we just shuttled the problem over to the file system, which now had to deal with thousands of small files on every request. Yes, this problem goes away if you configure an alternate cache backend, but most concrete5 sites don't do that. On web hosts with slow disk I/O this was a killer: hitting a site with no cache entries or out of date cache entries could take a really long time. Oftentimes the next access would be faster, but not always; the filesystem could be a significant bottleneck on shared hosts, and it was clear that caching was hurting as often as it was helping.
Cache Bigger Objects: Enter 5.6
We recognized this, so we tried to cache larger and larger objects in fewer and fewer cache entries. The idea was that if we could still use the cache but access it less often we'd get the benefits of the cache without as much of the lookup slowness.
This reached a head in concrete5 5.6.0. Our completely rebuilt permissions system was designed from scratch to be extensible, powerful and able to handle anything. Unfortunately, it requires quite a bit more database access to work with. Rather than cache each permission lookup, we ran all permission lookups every time a page was requested, and cached that in the page object. This worked but performance was significantly degraded for sites that had caching off. And when I say significantly degraded, I meansignificantly degraded. I think at one point I benchmarked around 700-800 queries running on a simple page when caching was disabled.
This was where we were in the Fall of 2012: concrete5 was more extensible than ever, but also slower than ever. Sites would fail due to slow disk I/O and the response was either "disable your cache" or "enable your cache." Disabling the cache would bloat the database queries by the hundreds, many of them duplicates. It was clear that caching was causing more problems than it was solving.
At this point, we had several layers of performance mechanisms:
- The in-memory cache: powered by a simple array, the in-memory cache (aka "CacheLocal", if you're curious) is a way of storing a requested object in memory, so that it wasn't retrieved from the file system if it had already been retrieved once in a given request.
- The basic cache: powered by Zend Cache, this was the object-based caching, usually performed by the file system, that I've described above.
- Block caching: introduced in 5.4.0, block caching is a powerful way of preserving all that is great about blocks while allowing developers to mark ways that their blocks can be cached, in order to preserve their performance on render.
- Full Page caching: also introduced in 5.4.0, full page caching is a way to store the entire contents of a page and render it ideally without accessing the database.
- Overrides Caching: introduced in 5.6.0, overrides caching was our way of saving the state of your overridden file system to a Config variable, so that we don't have to do as many file_exists() checks throughout our code.
While these were all noble attempts at solving the performance problems of a complex and flexible application, they werealmost all failing:
#2 failed because too many items were being placed in the cache, the file system is slow on many web hosts, the item still has to be requested from the database and then written to the file system, and it just wasn't much faster to request from the file system than it was to get out of the database. Furthermore, by forcing more data to be written into the cache in order to create fewer entries, we actually did many more database queries to make #2 viable than we normally did. This killed performance especially when cache was disabled.
#3 was great in theory, but it used #2 as its backend, which meant all the problems that plagued #2 filtered down to #3. This was the same problem with #4 but in addition #4 suffered from firing far too late in the loading process. Practically, #4 never ran without still connecting to the database, which rendered its benefits minimal. Finally, #5 was a good idea in theory, but still required a database connection, and was confusing, because it couldn't be cleared by deleting files/cache/ directory in the filesystem its data was stored in a different way.
The Present: 5.6.1
It's no secret:we have great plans for concrete5 5.7.0. But these plans are somewhat contingent on concrete5 running everywhere, and running everywherewell. We need to work well on good servers, and work acceptably on even the slowest ones. We simply cannot afford and will not accept performance continuing to be such a problem.
So I set out to fix it. To do this, I decided to look at what worked, and throw out what didn't. The override cache worked and was a good idea, but storing its information in the database to get around the slowness of the cache was stupid. Full page caching was a great idea, and the options that triggered it were sensible. But relying on Zend Cache and firing late in the loading process? This was a bad idea. We had tried to make it so that controller actions and certain interactive items could still be used with full page caching. While this was neat in theory, it made full page caching much less effective, and most importantly just plain slower.
Block caching is a great idea. It occurred to me if we had had block output caching first, when the gallery problems had cropped up with the file block, we may never have implemented anything else.
What didn't work? The one-size-fits-all cache, using Zend Cache as a backend. At best, it still added uncertainty to concrete5 ("Something not working right? Clear the cache, I guess."). At worst, it degraded performance on first run and, for many sites, in perpetuity.
Let's be clear: Zend Cache is a fine piece of software. We will still continue to support its usage in our API and actively use it when interactive with Zend Framework components that use it (like Zend Translate). But shoving every object in there because we make too many database queries? That's like applying a bandaid to a cut with a pair of scissors.
You can currentlydownload a beta version of concrete5 5.6.1to see these changes in action. I would encourage you to do so, and to take it for a test drive on any web host. The cheaper the host the better.
What's been done? A lot.
We don't use the Cache anymore. It's still there and used for Zend Framework libraries, but rarely accessed anymore. I think the information about the dashboard picture of the day might use it but that's it.
We do use the in-memory cache. A lot. This is CacheLocal. The first time in a given request that a page, file, permissions record or other objects are requested, they are added to the local cache, and then subsequent requests for these items are retrieved from an array. This dramatically reduces database queries without incurring any disk I/O problems. Yes, you do occasionally run into odd cache-related bugs, and it could cause some problems for hosts with very restrictive memory settings, but these should both be rare, since CacheLocal doesn't persist across subsequent requests.
We do use block-level caching, for records and output, but we store this information in the database. We're already requesting data from these database tables, so adding another column to the mix and displaying its contents is trivial (and much faster than looking in the file system for Zend Cache entries.) All the same block caching settings that were valid pre-5.6.1 are still honored. The best part about block-level caching: we save block output and information when a block is saved, which means the data is already there for subsequent requests, leading to much less slowness on initial request for a given page.
We do use the Environment cache, but it's stored as a serialized object in files/cache/environment.cache. Clearing the cache will remove this file, as well as just deleting files/cache/ yourself.
We have updated our CSS cache to store files in files/cache/css/. This is used whenever a theme uses customizable stylesheets. Files are generated and referenced directly without going through concrete5.
We have completely rewritten the full page caching library. Full Page caching is completely modular and extensible. It fires very early in the request. If a site enables full page caching, and the pages are in the cache, a site can completely lose its database connection and it will still render as though nothing is wrong.
We have optimized and removed unnecessary queries. So many queries were structured in certain ways to appease the mighty Gods of cached objects, that when this was no longer a concern they can be rewritten. Page and File custom attributes are an example of this: we used to retrieve all attributes about a particular file or page when that object was retrieved, in order to cache as much in the object as possible. Now we only retrieve this information on demand.
The results speak for themselves. concrete5 5.6.1 is much more responsive on many web hosts. Yes, it still makes robust use of a database (non-logged-in users will see around 50-70 database queries rendering a standard starting point page that's not cached in the full page cache) but this is a massive decrease from 220.127.116.11. And its all being done without the cache adding extra disk I/O.
We are still actively exploring how we might speed up some core block types like Auto-Nav and the PageList class. This may require some creative, not-necessarily-object-oriented problem solving, but we're up to the challenge, armed with some of the lessons we've learned.
One Size Doesn't Fit All
The features we'd added for the sake of performance weren't all hindrances but their impact was lessened because we tried to funnel them all through the same approach (a file-based cache.) Now, although we have some slightly more idiosyncratic approaches to solving these problems, the results are much, much better.
Keep the Good Ditch the Bad
We've never been afraid to rebuild something in a better way.
Remember the Problem You're Trying to Solve
We should have realized we had a problem the moment we started trying to fix the cache. The cache isn't a feature of the software: the cache is means to an end (faster software!) Focusing on actually reducing database usage throughout the application in sensible ways rather than adding them and shuttling them into a cache once they've run is going to serve us far better.
Let us know what you think.
(Reposted from Andy's Blog)
It's no secret: concrete5 is a beefy application. We pack in a lot of dynamic features, have a very extendable code base, include large libraries from the Zend Framework and offer powerful permissions and design flexibility. In turn, sometimes site performance suffers, especially on cheaper web hosts.
Unfortunately, somewhere along the line, performance problems started to become more than just a nuisance and threatened to become ameme. We had to do something about them.
Its more important what you say no to in life than what you say yes to. When we went open source, we stared at google analytics every day and rejoiced when we passed 100 visits a day. Now we average 10,000 visits in a day and were the fastest growing open source CMS out there. In the early days we were just flattered that anyone cared, so wed bend over backwards to provide any service we could to anyone who asked. As time went on, we tried out different ideas and dumped a few that turned out to be mistakes or just past their prime (pro accounts, support incidents to name just two). Were banking on 5.7 being a pretty big deal, so to make sure were positioned correctly to support and promote it, were focusing on a few things:
- No more budget hosting. To stay the size I want we have to choose where to put our hosting expertise, and were focusing on the big boys. Theres plenty of budget hosts willing to do a great job helping people get started with concrete5. Were going to have the free hosting with ads at concrete5.com, and those ads are going to suggest you get a paid hosting account to get rid of the ads. I dont expect to make much of anything on the ads (who knows) but I expect to make a lot on the hosting signup commissions.This would be awesome for us as it would help me track ROI on marketing spends.
- Enterprise Support SLAs. Since launching the enterprise site weve already sold a few of these guys with great success.
- Training. This is the #1 priority for growing the project. Weve got to offer more training in more ways and get certification working. Were seeing huge system integrators looking for ways to train 1000 developers on concrete5, and what weve got today is only now starting to meet that need.
- Services. Were still helping a handful of clients with concrete5. We like having our feet in the water so to speak, as we believe it gives us insight into ways the tools need to improve. Theres an old sales adage of you can have it fast, cheap, and well done - now pick 1 (or 2). It strikes me that we should be expensive and perfectly done. The more we move in that direction, the more room there is for the community to move up and be reasonably priced, fast, well done.
- The marketplace. For 2-3 years I talked about very little beyond the marketplace when people asked us how we were going to make money. I always got inquisitive looks when I was speaking to well informed folks, but I was passionate about it nevertheless. I believe weve made a more powerful solution to the module/add-on problem than all of our competition has today. I believe the PRB and community curated aspect of our marketplace is a key component to our projects success, and will always remain so. So many of us have been burned by Drupal modules that break other modules, or Wordpress add-ons that just dont work. Bringing the barrier to entry up a bit and using Apples App store as a guide was a good idea and I have no regrets about it.
That being said, there are some real issues with it being the primary or only revenue model:
- Practically speaking, it has been flat for us for some time. Its just very dependable income each month but it never explodes and it doesnt seem to be that directly connected to our site traffic or demo signups. Practically the marketplace is not our primary revenue model today. Marketplace net profits cover of our monthly operating burn rate.
- We cant sell enterprise stuff there. People are outraged when we price an internationalization suite at $1,750 in the marketplace, but when I tell people through enterprise.concrete5.com that our enterprise suite costs $250k for an unlimited license no one blinks an eye. Moreover any enterprise that might want to grab a simple add-on out of there instantly has support issues. A large organization needs one point of contact for support issues and is willing to pay for that. Since we really cant promise failsafe support on add-ons we didnt create, even with the PRB, its often difficult for the big boys to see the marketplace as anything more than a nice prototyping perk.
- Its a turnoff for the moral high ground open source crowd. The fact that were MIT licensed instead of GPL is already a hard sell for some open source advocates, and then they look at our marketplace and exclaim that all the good stuff costs money! I can (and do) argue that actually about half the stuff in there is free and you can buy stuff for Wordpress/Drupal/Joomla too. We just took the effort to make a curated destination with support and refund policies, but it still rubs folks the wrong way at a glance - and that is what it is. If someone thinks you smell, it doesnt matter if you know you dont from their position, you smell. The idea of making something and selling it in the marketplace tends to appeal to the 1-2 man entrepreneurial cowboy webshop looking for different ways to augment their services revenue with something more dependable. Speaking as someone in that camp, I think thats great. Speaking as the leader of a open source project with the goal of being a ubiquitous building material for the web, I wonder if we may be letting ourselves get unfairly compared to more truly open source offerings out there by always saying the marketplace is our primary revenue concern.
- Our own add-ons are priced in a way to try to cover the costs of developing the core. What ends up happening is our add-ons arent as good as they could be, because revenue that should be going back into improving them gets funneled into time on the core. We have too many small things in there which require too much support expertise. It also makes tracking ROI on any marketing spend difficult. The process of discovering concrete5 to choosing to buy an add-on is too squishy and hard to track for me to justify spending dollars marketing concrete5, so today I dont. Id love to change that.
To address some of these issues, we want to change the way the marketplace works:
- 5% gross of any sale should go to a non-profit the customer picks from a list on checkout. Ive wanted to do this from day one but foolishly never did. Theres a clothing store here in Portland called Buffalo Exchange that gives you a wooden coin to put in one of three boxes if you dont need a bag. The boxes are each for a different charity that they cycle out monthly or so. They keep them very safe (no politics, religion, etc - stuff like shoes for kids or saving dogs). I believe that by doing this we will address two branding issues: First I believe peoples expectations about support from our add-on developers will be more reasonable. Second I believe the I cant believe this stuff costs MONEY crowd will have the sense to keep their mouths shut, or risk looking very cheap.
- 2% gross of your purchases above $1,000 should go back to customers as rewards credits. Use them for more marketplace purchases or just cash it out and buy a beer. This gives me something to say to the shops looking for a reseller/affiliate/commission type deal.
- To make those happen, wed ask marketplace developers to give up a 30% cut instead of the 25% we ask today. ;) Seriously though, this is one of those places where making it up on volume actually works. These changes and the ecosystems changes will make me feel comfortable spending money marketing concrete5 (which doesn't happen today.)
- Were not going to call it a Marketplace any more. It's simply the community, as we now will have concrete5.com the free hosting site as well.
- Our Add-ons - were going to start selling fewer things in the marketplace. Some of our stuff should just be free at this point (superfish??). Some of our stuff will be rendered obsolete with 5.7 (discussions and calendar). With eCommerce we plan on having a basic free version, a $95 version that is a bit simpler to use than what we have today but is somewhat comparable, and a $295 version that includes complex product configuration and other goodies. Were going to start selling some themes as well.
We are speaking to many disparate audiences with one site at concrete5.org. Adding the enterprise.concrete5.com site has been very helpful, and was a no brainer we should have done a long time ago. Enterprise CMS solutions are licensed in the low-mid 6 figure range, so it just doesnt make sense to put any of the tools were working on for large clients into our marketplace at concrete5.org. Same goes with the support SLAs that are important.
A similar problem exists for consumers vs designers & developers at concrete5.org. If Im a small business comparing concrete5 to the DIY web builder market it just doesnt make as much sense at concrete5.org as it could. Sure theres an instant setup link buried at the top of the page, but that is just a demo that we cant easily upsell into a hosting account.
To clear it all up:
- Redesign concrete5.org to feel more like a developer/designer community (bit cleaner design with a tad less whimsy). Improve the features around finding a community member and posting a job, but keep all that free. Focus more on documentation, including general cleanup and improved API documentation.
- Launch concrete5.com to be one click install of concrete5 for free at Yoursite.concrete5.com. Much like wordpress, offer free hosting on a cloud server with ads for ever. Want to lose the ads? Click a built in promo to one of our hosting customers and have your site packed up and moved automatically (see business changes).
- Make concrete5.org, .com, and the app itself work with Facebook and Twitter authentication. Also make 1-click installs at concrete5.com automatically connected to concrete5.org project pages without you having to do anything. Getting a site on concrete5.com would automatically make or use a user account on concrete5.org. Theres just far too many logins required for people to understand today. Under the hood we kind of need it to work that way, but theres no reason we cant take some of the pain out of setting things up for the majority of new folks.
- Were debating just making concrete5.org a new site. With the new Conversations model weve got some real issues with our forums here (which are a one off). Were also starting to think that some of the legacy content in this site is going to be irrelevant with 5.7. Cleaning house is always appealing, and building a new site we can test in a sane way sounds nice too. Theres certainly no perfect answer here, but were entertaining any option at this point.