Hello everybody.

Work continues here, as I'm trying to see whether the problems we were having with wheezy are truly gone or not. We're staying on squeeze right now for the time being until I can find the problem or understand what was going on with the weirdness in the drafts system when I put one wheezy webhead into rotation. This isn't a huge problem, but it is blocking some future improvements.

The current theme of my work here is still within the engine of the site. It's both what keeps us alive and holds us back. For about a year, I've searched for a magic bullet, something that I could start to pull out of the code to refactor it so that I can introduce a new design and stronger features based upon the years of ultra cool data and contributions that we have. It's tough, but I've realized now that there is no magic bullet. I've also struggled with trying to figure out what piece of the code to go after first. To be honest with myself and with the users, I've been kind of lost with it. There are literally so many thing that need polish and improvement that it's tough to figure out where to start off with it. After literally a year of small fixes (also having a child), I think I've found where I need to start refactoring, and it's a long road uphill.

The first piece on my TODO list is to fix container multi-inheritance. That may sound bizarre, but it's actually one of the odd features that makes the site unportable. We have a problem where a lot of our application code is in the display layer of the database, which I'd love to strip out and push into our perl libraries. It's a tough problem; I basically need to "port" every htmlcode and container nodetype into something that can live outside of the database, and migrate the display piece of the site into a templating language. Before I can do that however, we need to consolidate application logic so that abstracting things out into controllers is a bit more doable. Right now the webpages make calls to the database inside of the display layer, which is maddening. The internal libraries (especially HTML.pm) are a mess, but at least they consolidate code that doesn't require an eval() to get to. It also makes debugging stack traces insanely hard to figure out where things are dying; making the wheezy transition a lot harder than it would be otherwise.

There is however hope. Here's what the current hit list is:

    In order:
  • Remove the last of the code from general container that handles app logic. That part is basically done as of this morning. Vote code has been consigned to the ghetto in HTML.pm, and that leaves only the container switches there. We did our IP logging and vote refreshing inside of there which was weird. This is an organizational artifact from people not having access to the direct source repository, but only being able to make changes to super core templates on the site.
  • Move the mobile container selection code out, and simply overload the standard zen containers to be mobile aware. This is a hack; a brutish, ugly hack. It will however keep us from modifying $CURRENTNODE, which I think is a pseudo-node.
  • Get rid of the general container. This is actually kind of a strange problem because it used to be so hardcoded into the site. It's going to take a lot of testing to make sure that it doesn't break anything.
  • Remove the themesetting nodetype; kill off alternate themes for classic and jukka. Inside of the genContainer stuff, we create a pseudo-node to represent that container, and that causes inconsistent handling with regards to creation of that resource. It's a hash that's structured differently, and we also taint it on very pageload. Pseudo-nodes are an abstract concept that means "any node that doesn't have it's final ref listed somewhere in $DB->{cache}". Anything that doesn't live in the nodecache requires us to go to the database, and it causes a mildly expensive getNodeWhere call several times per pageload. It's part of what makes our site susceptible to bot attacks.
  • Migrate the draft system over to using not a different nodetype. This is super snarly, but is one of the last things that is holding us back from making the output of getNode be a blessed object under which we can store subroutines and start to consolidate our app logic. This is going to be a big change, and hopefully mainlining some of the application logic that currently lives in htmlcodes will help us greatly. The way I'm going to need to do that is to develop a "delegation" system inside of Everything::HTML to push things over to Everything::HTMLCode or some such, if we look up an HTMLCode, but that other object "->can" handle it. We will pass the Everything::HTML globals into each call in a specified order, and that will start to let us properly scope those routines. It is no performance gain however; we already compile the hell out of those objects.

Jul 7

Today I've moved the vote code out of general container as I mentioned in the first point. Let me know if you don't get votes tomorrow. I have tested the change and it appears seamless, but again it's a slightly deep change. I'm worried a bit about AJAXy requests, but I think it's in the right place in the code.

Made it so that I can ban per-UserAgent if I need to

Jul 8

Abstracted away the isMobile check so that I can use it in other places and have it be consistent. Replaced it in set_mobile superdoc and general container

Straightened the way that general container works, so that it's now entirely a passthrough:

Themes are dead.

This means that while the selector might be there for a few more days, I am winding down the tendrils of the theme system. There is only one theme now, Zen, and we are removing the other ones from the system. The reasoning for this is long and documented, but the nickel tour is that there are literally five layouts (not styles, full-on layouts) within the system (jukka, classic, default, ekw, zen) since this site basically predated CSS. Since there's a way to do this now in a completely sane way that is supported by browsers, mobile, etc, we are moving to that model. The great work that people did with zen has put us a huge way there, so I am cleaning up the code so that we can start in on a new layout.

Styles are fine.

To be clear, Themes live on the Advanced tab in Settings, in the Unsupported section down at the bottom. If you are using, say, Bookwormier, or Jukka Dim Emulation, those are fine, and totally great. That's the kind of theme-ing the site should be doing; it's the same internal code path, just a different coat of paint in your browser.

Check out Style Defacer or Theme Nirvana for bits of new themes

Jul 9

Cleaned up 96 htmlpages that descended from general container or were junk. This is in preparation for deleting general container

Deleted general container. Old node, id 19. Goodbye old man, you served us well.

Next up is the duplicate display path cleanup. There are 128 distinct display code paths in the database. This is a combination of nodetype to displaytype. This used to be a 3-way relationship, with theme being the third variable. Now that we aren't considering themes, we can reduce the super common paths. Here's our targets:

mysql> select * from (select h.displaytype,n.title,count(*) as num from htmlpage h left join node n on h.pagetype_nodetype = n.node_id group by h.displaytype,n.title) a where a.num > 1;
+-------------+----------+-----+
| displaytype | title    | num |
+-------------+----------+-----+
| display     | document |   2 |
| display     | e2node   |   3 |
| display     | superdoc |   2 |
| display     | user     |   3 |
| display     | writeup  |   4 |
| displayfull | e2node   |   4 |
| edit        | superdoc |   2 |
+-------------+----------+-----+
7 rows in set (0.01 sec)

By my math, that's 13 more pages that can be destroyed. Once those are gone, we can then take a look at the outstanding containers that are not in any htmlpage-container path, and remove those. Once that's done, we can start to delegate the now mod_perl-safe htmlcodes to hard subroutines, and start to pull those out of the database and rescope them to remove the globals. Each one is also grep-able to see if we're using it or not. Once that work is done, we will then have the minimized code execution path to start working on for improvement. It also allows for optimizations in our display functionality; the most expensive piece of code in our codebase currently. I can't say enough how huge this is; this is the architecture change that will allow us to modernize the site.

July 10, 2014

I've undertaken an optimization in HTML.pm whereby we don't consider theme selection in the container search. This should be a page speed up, albeit a probably unnoticeable one.

Next: removing all instances of $THEME in the codebase

Jul 11


TODO: Wind down the variable keys inside of the zen theme so that it too doesn't matter anymore. This will uncover more places where it is used. That can be pushed into the general settings.

TODO: Remove the announcements bit from stylesheets; it was in the mobile layout, and is a no-op. I've removed it

TODO: Wind down the nodelet container concept. The only nodelet container is zen.

TODO: Delegate all of the htmlcodes

TODO: Fix the htmlcode page so that edev can see the code, even if its changed

TODO: devparselinks is hosed.

Jul 12

We are now at the minimalist container subset. From here, we can start the refactor. Htmlcodes are next.

Jul 13

Light day, since I'm trying to do a change a day here, but listcode now shows when a node has been delegated. See, like displaytable for an example if you are in edev

Jul 14

Delegated two more htmlcodes

July 15

For some reason, debates forgot that they had a grouptable. Ugh

Fixed the listcode htmlcode to work properly now for delegated htmlcodes.

Put in an optimization with the htmlcode delegation that's a bit experimental; it was calling the htmlcode both the new and the old way. It should now be doing it only once.

Delegated and un-cached the code in displaydebatecomment

July 16

July 17

Removed the parts of Advanced Settings involved with changing themes.

Removed some THEME keys to make grepping the last of its tendrils out easier

Jul 18

Continued down the road towards "nodes-as-perl-objects", and figured out that with the previous UNIVERSAL::isa work, it looks like it's just an invisible drop-in inside of NodeBase. Three places: getNode, getNodeById, and getNodeWhere. As I continue to refactor things, it's going to continue to get easier to push functionality into nodes themselves, write tests, and make sure that the site continues to work the way we expect it.

Jul 19

Made utf8 the default and removed it from the global configuration options

Made some modifications to the random nodelet items so that they can work in the test environment where there aren't any writeups.

Jul 20

Fix to zenwriteups for the test environment.

Jul 21

Started the process of delegating opcodes, largely because publishdraft was having errors under squeeze and opcodes are nigh-undebuggable. This means that errors in these opcodes will eventually bubble their way into the logs, making things like the squeeze transition findable. However, to pare things down to bare execution, I ended up having to remove the transactionWrap piece from publishdraft.

July 22

July 23

Fixed two small things; one with the way new experience was being shown, the other was with debate comments not working due to a bad reading of some code in the squeeze compatibility sweep.

July 24

    Deleted
  • legacyweblog
  • newwriteups2
  • isZen
  • displayuserinfo (Now only using zendisplayuserinfo)

Starting to clean out the isZen htmlcode

Jul 25

Moved the deployment to add a "default_style" parameter to Everything::CONF, so we can move that away from the $THEME hashref. Made that update to zen stdcontainer

TODO: Clean out Settings

July 26

July 27

Found out that there are two 'isSuspended'-s. One is an htmlcode, which was used in two places, and the other was a hard HTML.pm entry. I've removed the htmlcode piece: room maintenance create, Create Room, which fixes the ability to create rooms. Deleted isSuspended.

July 29

July 30

July 31

Edited chatterlight to remove the isEkw stuff. Killed off isEkw