Hello everybody.
Work continues here, as I'm trying to see whether the problems we were having with wheezy are truly gone or not. We're staying on squeeze right now for the time being until I can find the problem or understand what was going on with the weirdness in the drafts system when I put one wheezy webhead into rotation. This isn't a huge problem, but it is blocking some future improvements.
The current theme of my work here is still within the engine of the site. It's both what keeps us alive and holds us back. For about a year, I've searched for a magic bullet, something that I could start to pull out of the code to refactor it so that I can introduce a new design and stronger features based upon the years of ultra cool data and contributions that we have. It's tough, but I've realized now that there is no magic bullet. I've also struggled with trying to figure out what piece of the code to go after first. To be honest with myself and with the users, I've been kind of lost with it. There are literally so many thing that need polish and improvement that it's tough to figure out where to start off with it. After literally a year of small fixes (also having a child), I think I've found where I need to start refactoring, and it's a long road uphill.
The first piece on my TODO list is to fix container multi-inheritance. That may sound bizarre, but it's actually one of the odd features that makes the site unportable. We have a problem where a lot of our application code is in the display layer of the database, which I'd love to strip out and push into our perl libraries. It's a tough problem; I basically need to "port" every htmlcode and container nodetype into something that can live outside of the database, and migrate the display piece of the site into a templating language. Before I can do that however, we need to consolidate application logic so that abstracting things out into controllers is a bit more doable. Right now the webpages make calls to the database inside of the display layer, which is maddening. The internal libraries (especially HTML.pm) are a mess, but at least they consolidate code that doesn't require an eval() to get to. It also makes debugging stack traces insanely hard to figure out where things are dying; making the wheezy transition a lot harder than it would be otherwise.
There is however hope. Here's what the current hit list is:
In order:
- Remove the last of the code from general container that handles app logic. That part is basically done as of this morning. Vote code has been consigned to the ghetto in HTML.pm, and that leaves only the container switches there. We did our IP logging and vote refreshing inside of there which was weird. This is an organizational artifact from people not having access to the direct source repository, but only being able to make changes to super core templates on the site.
- Move the mobile container selection code out, and simply overload the standard zen containers to be mobile aware. This is a hack; a brutish, ugly hack. It will however keep us from modifying $CURRENTNODE, which I think is a pseudo-node.
- Get rid of the general container. This is actually kind of a strange problem because it used to be so hardcoded into the site. It's going to take a lot of testing to make sure that it doesn't break anything.
- Remove the themesetting nodetype; kill off alternate themes for classic and jukka. Inside of the genContainer stuff, we create a pseudo-node to represent that container, and that causes inconsistent handling with regards to creation of that resource. It's a hash that's structured differently, and we also taint it on very pageload. Pseudo-nodes are an abstract concept that means "any node that doesn't have it's final ref listed somewhere in $DB->{cache}". Anything that doesn't live in the nodecache requires us to go to the database, and it causes a mildly expensive getNodeWhere call several times per pageload. It's part of what makes our site susceptible to bot attacks.
- Migrate the draft system over to using not a different nodetype. This is super snarly, but is one of the last things that is holding us back from making the output of getNode be a blessed object under which we can store subroutines and start to consolidate our app logic. This is going to be a big change, and hopefully mainlining some of the application logic that currently lives in htmlcodes will help us greatly. The way I'm going to need to do that is to develop a "delegation" system inside of Everything::HTML to push things over to Everything::HTMLCode or some such, if we look up an HTMLCode, but that other object "->can" handle it. We will pass the Everything::HTML globals into each call in a specified order, and that will start to let us properly scope those routines. It is no performance gain however; we already compile the hell out of those objects.
Jul 7
Today I've moved the vote code out of general container as I mentioned in the first point. Let me know if you don't get votes tomorrow. I have tested the change and it appears seamless, but again it's a slightly deep change. I'm worried a bit about AJAXy requests, but I think it's in the right place in the code.
Made it so that I can ban per-UserAgent if I need to
Jul 8
Abstracted away the isMobile check so that I can use it in other places and have it be consistent. Replaced it in set_mobile superdoc and general container
Straightened the way that general container works, so that it's now entirely a passthrough:
Themes are dead.
This means that while the selector might be there for a few more days, I am winding down the tendrils of the theme system. There is only one theme now, Zen, and we are removing the other ones from the system. The reasoning for this is long and documented, but the nickel tour is that there are literally five layouts (not styles, full-on layouts) within the system (jukka, classic, default, ekw, zen) since this site basically predated CSS. Since there's a way to do this now in a completely sane way that is supported by browsers, mobile, etc, we are moving to that model. The great work that people did with zen has put us a huge way there, so I am cleaning up the code so that we can start in on a new layout.
Styles are fine.
To be clear, Themes live on the Advanced tab in
Settings, in the
Unsupported section down at the bottom. If you are using, say,
Bookwormier, or
Jukka Dim Emulation, those are fine, and totally great. That's the kind of theme-ing the site should be doing; it's the same internal code path, just a different coat of paint in your browser.
Jul 9
Cleaned up 96 htmlpages that descended from general container or were junk. This is in preparation for deleting general container
Deleted general container. Old node, id 19. Goodbye old man, you served us well.
Next up is the duplicate display path cleanup. There are 128 distinct display code paths in the database. This is a combination of nodetype to displaytype. This used to be a 3-way relationship, with theme being the third variable. Now that we aren't considering themes, we can reduce the super common paths. Here's our targets:
mysql> select * from (select h.displaytype,n.title,count(*) as num from htmlpage h left join node n on h.pagetype_nodetype = n.node_id group by h.displaytype,n.title) a where a.num > 1;
+-------------+----------+-----+
| displaytype | title | num |
+-------------+----------+-----+
| display | document | 2 |
| display | e2node | 3 |
| display | superdoc | 2 |
| display | user | 3 |
| display | writeup | 4 |
| displayfull | e2node | 4 |
| edit | superdoc | 2 |
+-------------+----------+-----+
7 rows in set (0.01 sec)
By my math, that's 13 more pages that can be destroyed. Once those are gone, we can then take a look at the outstanding containers that are not in any htmlpage-container path, and remove those. Once that's done, we can start to delegate the now mod_perl-safe htmlcodes to hard subroutines, and start to pull those out of the database and rescope them to remove the globals. Each one is also grep-able to see if we're using it or not. Once that work is done, we will then have the minimized code execution path to start working on for improvement. It also allows for optimizations in our display functionality; the most expensive piece of code in our codebase currently. I can't say enough how huge this is; this is the architecture change that will allow us to modernize the site.
July 10, 2014
I've undertaken an optimization in HTML.pm whereby we don't consider theme selection in the container search. This should be a page speed up, albeit a probably unnoticeable one.
Next: removing all instances of $THEME in the codebase
Jul 11
TODO: Wind down the variable keys inside of the zen theme so that it too doesn't matter anymore. This will uncover more places where it is used. That can be pushed into the general settings.
TODO: Remove the announcements bit from stylesheets; it was in the mobile layout, and is a no-op. I've removed it
TODO: Wind down the nodelet container concept. The only nodelet container is zen.
TODO: Delegate all of the htmlcodes
TODO: Fix the htmlcode page so that edev can see the code, even if its changed
TODO: devparselinks is hosed.
Jul 12
We are now at the minimalist container subset. From here, we can start the refactor. Htmlcodes are next.
Jul 13
Light day, since I'm trying to do a change a day here, but listcode now shows when a node has been delegated. See, like displaytable for an example if you are in edev
Jul 14
Delegated two more htmlcodes
July 15
For some reason, debates forgot that they had a grouptable. Ugh
Fixed the listcode htmlcode to work properly now for delegated htmlcodes.
Put in an optimization with the htmlcode delegation that's a bit experimental; it was calling the htmlcode both the new and the old way. It should now be doing it only once.
Delegated and un-cached the code in displaydebatecomment
July 16
July 17
Removed the parts of Advanced Settings involved with changing themes.
Removed some THEME keys to make grepping the last of its tendrils out easier
Jul 18
Continued down the road towards "nodes-as-perl-objects", and figured out that with the previous UNIVERSAL::isa work, it looks like it's just an invisible drop-in inside of NodeBase. Three places: getNode, getNodeById, and getNodeWhere. As I continue to refactor things, it's going to continue to get easier to push functionality into nodes themselves, write tests, and make sure that the site continues to work the way we expect it.
Jul 19
Made utf8 the default and removed it from the global configuration options
Made some modifications to the random nodelet items so that they can work in the test environment where there aren't any writeups.
Jul 20
Fix to zenwriteups for the test environment.
Jul 21
Started the process of delegating opcodes, largely because publishdraft was having errors under squeeze and opcodes are nigh-undebuggable. This means that errors in these opcodes will eventually bubble their way into the logs, making things like the squeeze transition findable. However, to pare things down to bare execution, I ended up having to remove the transactionWrap piece from publishdraft.
July 22
July 23
Fixed two small things; one with the way new experience was being shown, the other was with debate comments not working due to a bad reading of some code in the squeeze compatibility sweep.
July 24
Deleted
- legacyweblog
- newwriteups2
- isZen
- displayuserinfo (Now only using zendisplayuserinfo)
Starting to clean out the isZen htmlcode
Jul 25
Moved the deployment to add a "default_style" parameter to Everything::CONF, so we can move that away from the $THEME hashref. Made that update to zen stdcontainer
TODO: Clean out Settings
July 26
July 27
Found out that there are two 'isSuspended'-s. One is an htmlcode, which was used in two places, and the other was a hard HTML.pm entry. I've removed the htmlcode piece: room maintenance create, Create Room, which fixes the ability to create rooms. Deleted isSuspended.
July 29
July 30
July 31
Edited chatterlight to remove the isEkw stuff. Killed off isEkw