On April 21st, 2014 I started a campaign on Indiegogo.com to finance some work on the routing code for Joomla. (You can find the campaign here) The campaign ended on May 25th, 2014 and it already was successfull on April 25th. This is the place where the progress is being documented and where the contributors are listed. If you have any questions, please feel free to mail me via the adress in the Impressum.
Seems like I start all my postings here with "It's been a long time since I last wrote here" or something similar. This project has been really long, it was really complicated and it was often enough really hard to pull through.
After almost two and half years as a crowdfunding project and after 7 years in the making, the last code changes that were supposed to go into the Joomla core have been accepted both for Joomla 3.7 and 4.0. I'm so happy!
Today three important Pull Requests were merged by the Joomla team, one closing a bug in non-SEF mode, one cleaning up routing further in Joomla 4.0 (thus completeing the proper implementation for this in 4.0) and the last one making it possible to remove IDs from URLs in Joomla. With that and after around 500 hours of coding, explaining and defending, the work for the Joomla project in this crowdfunding project is finished.
I still have to provide you with perks and fullfill that part of the campaign, but with these things now finally under wrap, I can also now focus on this and move forward on this front. Thank you for sticking with me through this long time.
But now lets celebrate that we finally got to this point. I know I will.
Happy new year folks. It's been a crazy one for me. I had to move, I married, I did this crowdfunding campaign... I hope that you had at least as good a 2014 as I had.
But you are not here for some best wishes, but for an update regarding the campaign and routing in Joomla. In that regard I have quite a bit of good news for you. These are the Pullrequests, that I made production ready in the last few weeks:
Especially the last one is important, since this one finally implements all the behavior of the core routers in one easy-to-use rule. Most of these Pullrequests have not only the actual code, but also full code coverage via unittests in them. You might be wondering why the PRs contain so many changed files and seemingly the same changes all over, but I can guarantee you, that this is intentional. The changes that I did are building upon each other, so the changes in #5604 require the changes in #5446 and #5444. So the PRs are done in a way that when the first PR is accepted, I will update the others and the changes that have been accepted will vanish from the other PRs.
So now that the code is ready, this could go into Joomla 3.4.0, depending on the PLT, but more importantly depending on you. To get this into Joomla core, we as a project need more feedback on these Pullrequests. They need to be tested and a code-review needs to be done and developers have to take a look if they think this whole thing is sane. So please spread the word and help out yourself to make this happen.
just trying to keep everybody in the loop. Since my last update 12 days ago I did a few more PRs and 2 of those are routing related. Not only that, but these are major ones, which are the bread and butter of this whole project and which are the parts that will help component developers out there to write high quality component routers in no time.
The first one is the PR with the number #5444: Routing: Implementing JComponentRouterAdvanced
This PR implements a new base class for component routers that allows for rule based routing. It has a lot of abstraction in it. Most of it is described in the PR itself, so please read up and comment there.
The second one is the PR with the number #5446: Routing: JComponentRouterRulesMenu to find right Itemid
This implements the first router rule, which basically makes the *HelperRoute classes obsolete. In that PR in the testing instructions, you can see how your router would have to be changed to take advantage from this. The code that I've provided there will also be the code that in the end will make your router. Everything else can be removed.
I'm currently really busy, but I thought I'd throw that update out there and hopefully sweeten your holidays. :-)
Again its been quite some time since my last update regarding this crowdfunding campaign, but I have good news for you all out there. The project is progressing and we are reaching points where actually something significant for everybody is happening. But before I get to the good parts, I wanted to apologise again for the totally screwed up schedule. I was hoping to get this all done a lot quicker and have this project be finished almost 6 months ago, but I simply miscalculated the necessary work by several orders of magnitude. Each step, be it analyzing the existing code, writing new code (in bite sized pieces that can be digested by the project) or getting that code tested by others and convincing people about the direction this is going and last but not least merging the change in, took infinitely longer than I hoped. I hope that you can accept this apology when you see the work that has been done in the last 1.5 months.
First of all, let me list all the PRs that resulted directly or indirectly from this campaign since my last update:
- Fixing canonical in SEF plugin - merged
- Removing unnecessary run of JRouter::parse() - merged
- Routing: Implementing an interface for component router rules - merged
- Finder: Fallback for routing is called with wrong parameters - merged
- Optimizing JComponentHelper::getComponent - merged
- JRouterInstallation::parse() made compatible with the interface - merged
- Initial codereview of languagefilter plugin - merged
- Routing: Adding application and menu objects to component routers - merged
- Routing: using the dependency injected $app and $menu in the core routers - merged
- Updating JComponentRouterRulesInterface - merged
- Tags: Fixing routing
- Language: Refactoring the language system to simplify the code
- Implementing a tree node interface
- Routing: Implementing routing stages
- ContentHelperRoute::getArticleRoute needs to be called with catid and language
- Routing: Discover Itemid in Router for com_content
- Routing: Discover Itemid in Router for com_contact
- Routing: Discover Itemid in Router for com_newsfeeds
What do these links mean?
Each of these links is a contribution that I made to the project since the last update mid-October and that is related to the URL routing in Joomla. Every link with the prefix "Routing:" is directly related to the campaign that you funded, while the others are basically bugs that I found on the way to writing the new system. Most of the other bugs are related to issues where our current API was not correctly used in Joomla itself or overly complex code was used.
There is however one PR that I want to draw special attention to: Language: Refactoring the language system to simplify the code This PR not only cleans up the multi-language implementation in Joomla greatly, it also allows to do some major change inside of Joomlas routing. One of the main problems in Joomlas routing right now are the *HelperRoute classes, like the ContentHelperRoute class. These classes are doing something that actually should be done by the router directly, namely to calculate the right Itemid. The Itemid is a major part of the URL and defines the menu item that the URL should point to and thus the modules that should be displayed on the page (and several other things, but that would go too far in this post). Sometime during Joomla 1.5, these *HelperRoute classes were introduced and have been a problem ever since. First of all, you have to correctly use them everywhere that you are creating a URL (Which you can see here, seems to be a problem on its own already: ContentHelperRoute::getArticleRoute needs to be called with catid and language) and second of all, the router does not know about those classes. They are arbitrarily named and called BEFORE the router (which begins when JRoute::_() is called) and their return value (normally something like this: index.php?option=com_content&view=article&id=42:galaxy&catid=23&Itemid=15) is handed over to the router. Now, if you have such a URL in your article, Joomla will properly convert it to the "right" URL. If you however change the menu item in some way, that link will not be the right link to that article anymore. If the link will work or not, is mostly luck. Anyway, what I'm trying to say is, that the Itemid should be dynamically calculated by the component router. With the Language: Refactoring the language system to simplify the code PR, it is now possible to move that part from the HelperRoute classes into the respective component router, like you can see here:
- Routing: Discover Itemid in Router for com_content
- Routing: Discover Itemid in Router for com_contact
- Routing: Discover Itemid in Router for com_newsfeeds
This will make sure that all your links across your site have at least the right Itemid for these 3 components. (The rest of the core components will follow shortly) This will also soon be changed into generalized code so that this code does not have to be added to each and every component separately, but can be used by all components in Joomla, be it core components or third party extensions. This will then also be the first component router rule that shows the new rules based component router system and which implements the respective interface: Routing: Implementing an interface for component router rules and Updating JComponentRouterRulesInterface
Your question will most likely be, why I did not write that component router rule directly and instead am creating yet another interim solution. At the time of writing this post, Joomla 3.4.0 is close around the corner and I wanted to get this change into the system as soon as possible. Unfortunately, the base-class that would implement the rules based component routers is not ready yet. The code is written, but I'm not happy with it yet and have to give this some more thought. You always have to remember that we will have to live with every class, method and function and their respective behavior that I'm introducing now until at least 4.0. So when I screw up here in some aspect, we might even go further down the backwards compatibility hell that we are already in.
How can I help?
One thing that stalls development the most is the lack of testing. I, and the Joomla project in general, need more people that test the existing code-proposals. So if you want to support this effort further, please look at the link list above and test those proposals that have not been marked as "merged" yet.
What is with my contribution perk?
When you donated to this campaign, you selected something that you wanted in return, be it the plugin to get these changes in Joomla 2.5 or a router for your component to use this new system. I can assure you, that I will honor each and every of the promises that I made and deliver the perks to you as soon as the project is in a somewhat stable phase. But as I described above, the schedule was to optimistic and it will still need more time.
What is the further plan?
I'm working hard on getting more and more code done and getting people to test the stuff that is there. Depending on how much more Joomla 3.4 is going to be delayed, some of the stuff that I described above could go into this version. Otherwise it will most likely be released with Joomla 3.5.
If you have further questions, feel free to write me a mail.
Its been 2 months since my last update and I have to apologize for not updating more often. While I'd like to keep you in the loop as much as possible, I also have to have something to report and that unfortunately has not been much lately. I'm still struggling with the application router and its backwards compatibility. So much in fact, that I have to change plans.
I've made a "mistake" during this development and that is that I first wrote the unittests for the current router and then implemented my code. In the meantime those unittests have been accepted and thus the current core router code is completely covered with unittests and future code improvements will benefit from that. At the same time this means that I can't weasel my own code in there if it doesn't pass those tests and that unfortunately is still the case.
The current router has a pretty erratic behavior. It takes a JUri object and you can modify it, while at the same time it expects some variables to be returned as associative array. At the end, some of the data that it extracted from the URI is in the JUri object and some is in the associative array that is returned. I tried to make this behavior consistent by having the router-rules not return anything and instead transform the JUri object step by step.
Consider something similar to an assembly line on a conveyor belt. At the first step of parsing a URL, the JUri object arrives and special characters are decoded, the domain and folder of Joomla are stripped out, etc. All this data is written back into the JUri object. At the next step, the menu item is decoded, again storing the Itemid in the original object and removing the decoded information. And so the object moves along that assembly line until the human readable URL is decoded into something that Joomla can understand. This will not be possible in a truely backwards compatible way in Joomla 3.x.
A break in backwards compatibility will maybe affect 1% of all installations, but that would still mean several tens of thousands of Joomla systems, which makes this a no-go. I know, I promised full backwards compatibility, but this will not be possible for the application router and I'm sorry for promising too much.
The good news is, that this does not affect the component routers side of things. Since component routers have been based on two magic functions and only rudimentary class support has been implemented by me in Joomla 3.3, we can improve those without much regard to backwards compatibility. This means that we can have shorter, simpler and proven routers in the next release of Joomla. I will provide PRs for this in the next few days.
You might ask yourself, why I haven't done the component routers in the first place and you are right. I have to apologize for wasting to much time on the application router and not working on the component router more. I approached this a bit too linear and first wanted to fix the application router before approaching the component router, which is invoked by the application router. I got a little carried away.
So to summarize all of this: Unfortunately the application router can't be implemented in Joomla 3.x without breaking backwards compatibility and thus breaking the rules of the project. We will have to wait for Joomla 4.0 to fix that part of Joomla. I might be able to provide an interim solution until then. The component routers (which you might find more interesting anyway ;-) ) will be able to be implemented in the next Joomla release if the project accepts.
As a sidenote, we also got 4 other bugfixes out of this project so far.
While my updates were sparse, I haven't been idle in the last few weeks. The current situation is that the new routing classes are done and the old behavior has been put into rule classes, however this code is not backwards compatible yet.
JRouter has been reduced from 29 methods to 10 (those 19 methods have been left in but deprecated), and JRouterSite has been reduced to 2 methods from formerly 12. While the interface has been kept backwards compatible, the behavior has changed and this is where the problems set in. Is it still backwards compatible when the methods don't have the same effect as before? Right now, when you set the mode of the router between SEF and RAW, it changes the URLs. With the proposed changes, there is no mode anymore. It is just a changing set of rules that is applied to URLs. So I'm struggling on if and how this is B/C.
There is also the issue of the interface of rules. In the past, build-rules did not have a return-value while parse-rules did and in the new system, this has been unified so that both don't have a return-value. I think I have a sufficient solution for that, but it needs more testing.
Despite these issues, I'd like to give you a little insight on how this works right now. JRouterSite still extends JRouter, but it does not really override any methods anymore. Instead JRouterSite defines a set of rules that are added when the router is instantiated. It also takes an app- and menu-object that it provides to the rules, so that URLs for the frontend could be build both in front- and backend, depending on if you are able to get proper app- and menu-objects. JRouter then only provides a build() and parse() interface, which executes the different rules on the URLs. So JRouterSite is mostly empty and JRouter itself is pretty lean, too.
For the rules, there are two ways to add them to the system. Solution #1 would be a callback, that you can hand over to the router. Solution #2 is simply a rule-name, like "sef", which gets expanded into "JRouterRulesSef". That class should implement the JRouterRulesInterface and up till now, all those rules are autoloadable. Each rule provides a build() and a parse() method, which take the router and the URI object as parameters and can then modify them to their liking. Both objects are handed over as reference, so that the methods don't need a return value. While I could implement the rules as abstract classes, I decided to work with instantiated objects here, since this allows you to do some preliminary work like caching the menu or similar stuff.
I will try to keep you posted more often and as soon as the code is presentable, I will push it into my cloned Joomla-github repository. Thank you all for your patience, I definitely have not forgotten you.
Hello to everybody and especially all the contributors!
I can happily announce that the first pull request has been created: https://github.com/joomla/joomla-cms/pull/3759
This PR introduces a large chunk of tests to document the current behavior of the core router classes and is the foundation to rewrite the system in a backwards compatible way.
In the future, I will keep you all posted on the progress of this effort here on this site. Considering the timetable: I'm lacking a bit behind, but all major roadblocks should be gone now and I should be able to devote most of my time to this project.
This is an alphabetically ordered list of all the contributors to this little project. Let me say "Thank you!" to all of them. If you want to have other information displayed here instead of only the nickname, please mail me at email@example.com. I will publish your name, company name, link to a website and a mailadress if you ask me to.
- 4Web Inc.
- Andreas Ruttkamp
- Dan Knauss
- David Jardin
- Digital Peak
- Dmitry Averin
- Dmitry Rekun
- Gary Barclay
- Ivo Apostolov
- Joomlapolis .com
- Kim Hjortholm
- Marko Dedovic
- Matt Thomas
- Matthew Philogene
- Octopoos Editor of SEBLOD
- OSTraining .com
- Paul van Jaarsveld
- Paulo Griiettner
- Pulsar Informatique
- Robert Went
- Roberto Segura
- Signum Product Solutions
- Simone Bussoni
- SoftForge Ltd
- SOHO Prospecting
- Tobs Bobs
- Valentin Despa
- Valentín García
- Viktor Vogel
- Webworker Berlin