I can see the beauty of mathematics…You can hold infinity in your hands

After some consideration I’ve decided to revive and rework a product I wrote about 5 years ago both for commercial reasons, and as a redesign and architecture exercise. I thought writing about my experience spelunking into my old code might be useful, so I’ve decided to document my self humiliation for posterity. Part 1 of this series will focus on digesting and assessing code I wrote five plus years ago as a much less experienced developer and planning an approach to attack modernizing the product, and getting it production ready again.

Background:

This product was a great test bed for assessing my overall skill improvement. The original came in five major parts.

1. Interaction with a native C++ driver in win32 for multiple input devices and displays
2. A multi-threaded near-real time data processing engine
3. A gui program responsive on multiple screens.
4. A data recording system based on MySQL that performed analysis and export.
5. A plugin system for adding new analysis engines to the program after installation.

The original system was written using C#, along with a managed C++ wrapper to the native win32 driver dll for the touch screen devices. It also consumed LGPL libraries for generating excel spreadsheets and interacting with a MYSQL database.

Initial Breakdown:

This code stands up surprisingly well for 5 years of disrepair. I have to credit Microsoft, other than the outdated installer code and a couple of broken reference links it translated directly from Visual Studio 2010 to Visual Studio 2015 without a hitch, including jumping 3 minor versions of .NET. Overall I am relatively pleased with my design choices in the original version. The code is pretty readable and I followed good modularity patterns so the various pieces of the code are kept fairly discretely. My early work to divide the modules properly is going to reap dividends as I should be able to do the rewrite piece by piece if I conform to the original interfaces.

On the downside, I was handicapped at the time both by my ignorance of C#’s concurrency mechanisms at the time, and the relatively new presence of async/await. I hand rolled all the code directly using the System.Threading.Thread interface rather than dispatching the work out to a thread pool, and thus I ended up manually using things like reset-events and concurrent queues to handle multithreading at a much lower level of abstraction than I’d use today. I’ll re-write and modernize this to take advantage of the better concurrency primitives in F#, and use the mailbox processors to represent the linearization of events for each screen rather than hand rolling it with queues like I did before

Also, lets just get it out there. The GUI was just plain atrocious looking. I’m going to take an initial stab at redesigning it, but might enlist the services of a better designer this around. I used windows forms, which was the mature choice at the time. However, because of the limitations of windows forms application my application could only actually handle one cursor any screen at any time. This meant I had to do an ugly hack and store the cursor position on the “main” screen and recenter it after any touch events on the auxiliary screens, which is way less than optimal. Now that WPF has matured, I’ll probably invest in redesigning the GUI for that framework which will allow me to more cleanly separate the input events from the various devices and cleanup the command and control code considerably.

Other things I notice. The analysis package in C# is pretty hackish. I’d like to rewrite it in F# for clarity, and consider exposing a read-only API directly from the application to R/Python/Matlab as an export so I don’t have to constantly hand-roll statistical analysis into the program. This will allow for better integration with my target customers workflow and save me from having to do a custom analysis engine for every target.

Plan of Attack.

1.  Pull the latest drivers for the hardware I’m using and re-write a more mature, and tested version of the managed wrapper. I’ll probably open source this, as I see no reason not to.
2. Re-write the core processing engine in F# with more clean abstractions for concurrency
3. Re-write the data storage engine to use SQL-Lite, rather than hardcoding the data model into MySQL.
4. Add plugins for writing the data to mongo, mysql, and flat export to json/s3
5. Rewrite the GUI in WPF with multi-cursor support.
6. Add a data analysis API for processing the export data in R/Python directly, as well as modernizing the .NET plugin architecture
7. (Maybe) repackage the installer and set up for delivery over the internet.

Technical debt is every programming team’s mortal enemy. It can slow development progress to a halt, leaving management wondering “what went wrong” as a formally productive engineering team suddenly hits the brakes and output seemingly grinds to a halt. For the non-techies, the term “technical debt” refers to the build up of messy, complex, poorly document, or fragile code inside a project or company. Most companies are aware of the primary source of technical debt: compressed schedules, and programmer laziness oversight. Good engineering teams employ multiple strategies to keep incidental technical debt under control. They have code reviews, encourage testing, documentation, and coding standards. These measures give team leads warm fuzzy feelings that we are doing our part to keep the technical debt under control. Sometimes we might even push for allocating time to clean up particularly egregious pieces of legacy code.

So it’s simple right? Follow the best practices, put good strong standards in place, get buy in from your peers and management, and you’ve eliminated the problem altogether right? Done and Done?

If only it were so simple. Beneath the surface of even the most gold-plated code base lurks a more insidious form of technical debt. The accumulation of good decisions! Engineers couldn’t be happier than being heads down, in the trenches cranking on technical problems. Minor missteps, not bad enough to trigger any of the standards slowly add up to be a big problem.

It’s so easy to fall into the “doing too many good things” trap. A quick win, I’ll just write a utility here that’ll handle this edge case. This new library doesn’t quite cover all the cases solution did, so we’ll run them side by side for now. The code is documented well, and up to standards, but little by little the complexity of the code base crawls upward in spite of engineering’s best efforts. The number of projects and folders explodes. There are utilities controlling which utilities can run. Frequent comments of “oh, yeah, let me dig up the link to the documentation, I set that up a month ago”. Before you know it, all these well intentioned, even well executed solutions have grown into a many headed hydra that is beyond the best engineer’s ability to grasp it all.

I’m here to suggest that you take a moment with some regular frequency, step back and look at the big picture. Surprisingly I find it’s can be bringing the team members up to speed that can illuminate this kind of description. Is the environment setup getting long? How many times do you have to explain “this is a little awkward but…”. How much trouble is it for new engineers to ramp up on the oldest parts of the code base? These kind of cues are flashing warning signs for the attuned developer.

Take the time to step back with some frequency (I’d suggest monthly, but this will vary upon the rate of change of your codebase). Pause your task list, pause your email and take a morning spent reviewing your system as a whole and get a feel for the various components. Be ruthless in looking for pieces to prune, combine, or refactor. Could we move this utility into our primary services api? Can we rip out this old library and change the code to use the newer solution we use in our second product? It’s so easy to get attached to our solutions, especially once they’re in code. They represent hours of our time and effort. Do your best to embrace the experimental nature of your job and prune the dead branches out of the tree before it becomes an impenetrable thicket.

Don’t let your desire to move fast and break things turn your good decisions drag you inevitably down the path of going full nuclear and re-writing.

I am an unabashed fan of type safe, or at least, strongly typed language. I got my formative training in programming and computer science at Carnegie Mellon which has a department full of type theorists. I spent a lot of time playing with ML-family languages, NJ/SML in college and F# in the beginning of my professional career. One might, (okay, a few friends would definitely) say I’m a bit of a snob when it comes to programming languages.

And yet, more an more of the time when I come across an idea for a new project, I find myself reaching into my toolbox, not for F#, Ocaml, or Haskell but rather for python. It’s crept up on me little by little. When first picked up python I have to admit it was with a sense of mild distate.  My wife works in biological science and needed a couple of cross-platform data wrangling scripts. None of the languages I knew well fit the bill and python seemed like an obvious choice, if only to branch out and learn something new. Early experiences with the language didn’t do much to change my opinion about python in general. Significant whitespace felt gross and inelegant. It felt like the language was trying desperately to hold my hand and protect me from myself. It felt clunky and awkward like a pair of borrowed shoes.

Little shoots of positives started creeping their way through the wall of my snobbery. It seemed like the library I needed was always available, ready to bolt in to my script. The mantra of “batteries included” seemed pretty built in to the standard library and I found that my “zero to done” time on these little side scripts plummeted. It was getting harder and harder to justify my distaste of the language and I started to get a feel for the places where it excels. Eventually my proficiency in the language improved and it started to bleed over into my fulltime work, where C# and Javascript rule the roost. For quick prototypes and one and done tasks I found myself reaching for python first. This was greatly magnified when I began doing more and more data analysis work and discovered the absolutely excellent scikit-learn.  I’d say at this point python has pretty thoroughly won me over. It’s my goto language for lightweight services or prototyping tasks to be sure. There are just too many solidly written libraries for me to choose anything else. The “get things done” philosophy wins out over my ideal languages preferences

That isn’t to say the language doesn’t have its share of warts. The syntax is still ugly, and there are a few limitations in the lambda syntax I could do without. Also, I’ve yet to find a web library I really love, although I have been experimenting with Flask and it’s reasonable so far. I also fell victim to the dreaded global interpreter lock (GIL) and made the classic misunderstanding of how interacts with multithreaded code. I think this is more a matter of  poor documentation but it bit me for a few hours threading CPU bound code before discovering this would result in zero improvement. This also means using additional cores on CPU bound tasks requires using clunky multiprocessing solutions which chew up memory, or dropping down into C which eliminates pythons productivity edge.

Still, despite these points of contention python has earned a place near the top of my toolbox. It’s become my trusted pair of channel locks I can always use to get the job done in a pinch, even if it’s rusty around the edges and a bit stiff to work with. I highly suggest you give it a whirl. You might end up pleasantly surprised.

I’ve had an epiphany about productivity, and finally managed to put a finger on a bad habit that’s been eating at me. I have a tendency to be a purposeless user of technology. It’s not a matter of me doing anything particularly wrong, it’s that I am, by nature, indecisive and risk averse when it comes to my use of my time.

When given an idle moment I could make a choice about how to spend the time. I could pick up the guitar, tackle a project around the house, hack on some code or spend some time playing a game I’d really enjoy. Unfortunately making that decision takes a bit of cognitive stress. I’d have to actually allocate that time and consciously chose to not do all the other things I’ve just mentioned. My lazy brain opts instead to take the path of least resistance and thus it’s much easier to flip open the laptop and scan hacker news for interesting tidbits, or refresh twitter for the umpteenth time while I “try to figure out what I should do”. Minutes become hours and then it’s suddenly time to crash for the night and I’ve had net gain zero from my little bit of free time.

It’s an insidious habit and one that’s particularly easy to self-justify. After all, I’m not idly flipping through meaningless facebook posts or watching meaningless television or movies (usually). My twitter feed is well curated with interesting people saying interesting things. Similarly, my google reader is full of articles that have real value. Unfortunately, the 100th article of the day gets me very little marginal return for my precious investment in time which in the long term is crushing my free time productivity. I’ve started to recognize that I have the same habit when it comes to consuming content. When I’m tired and want to veg out, I end up watching tired old favorite movies again and again and spending hours I’ll never regain.

Note that I don’t think there’s anything wrong with the technology I’m using. Netflix is awesome! Google reader coupled with Hacker News has been an astoundingly good resource for keeping myself current with trends in technology and has provided me with many, many useful ideas I’ve carried in to work. Social tools, email, and chat all fall into the same bucket. I’ve learned new things, and kept in touch with people who I’d otherwise have lost long ago. It’s not the tools I’m using and I really can’t even blame it on their addictive nature. The culprit is truly my lazy brain.

In the spirit of a new years resolution I’m making a pact with myself to become much more meaningful in my use of technological tools, and more meaningful in how I spend my time. The practical consequence of this is I’m making a pledge to myself to not sign in to any computer, smartphone, television, console or other gizmo without first doing an explicit mental check and clarifying the purpose. If I can’t express to myself the purpose for what I’m doing, I’m not going to sign in. When I complete my task, or session I’ll sign out, log off, etc and take another pause before I start the next task.

Obviously this is going to require some degree of self-discipline but I’m hoping that adding a mental check will be enough of road block that I can short-circuit my brain out of the path of least resistance and make it easier to choose to do something meaningful with my time. I think this will also give me a chance to put some useful restrictions on my consumption of “useful” content. I doubt if I’ll even notice the difference if I halve the time I spend per day reading the articles because I’ll simply filter out more of the irrelevant content. I can put this freed up time to good use, writing, hacking, or doing a thousand other more useful and enjoyable things.

I’m certain I’m not the only one afflicted with this form of NADD . I’m hoping maybe if you’re reading this it helps you reframe your own use of time. Maybe we can all start using our technological tools with a little more purpose and stop letting them nerd snipe our lazy brains.

I pitched a talk at BarCamp Charleston on problem solving.

Slides are here

I just finished watching the the excellent documentary Mr Rogers and Me about the life and work of Fred Rogers. I’ll be the first to admit that I’m an unabashed Mr. Rogers fanboy, and have been most of my life. If you don’t know who he is, take some time on wikipedia or watch him saving pbs or making a room full of soap stars cry. He was a man of universal acceptance and compassion. And yet, as remarkable as his person was, Fred Roger’s most lasting labor, and legacy, was his believe that television could be more than a vehicle for more than entertainment and the selling of products. He believed that it was medium through which a child could learn, and find acceptance and comfort from the dark times in their lives.

As I sat there watching I couldn’t help but wonder, where is the “Mr. Rogers Neighborhood of the internet”. Now I’ve literally grown up in parallel with the internet. I watched it transform from a  toy of the technical elite into the predominant platform through which people communicate. I’ve seen the rise of AOL instant messenger, Facebook and Twitter. Yet for all these accomplishments where is our “Neighborhood”? A place that is safe, secure for children. A place they can turn when they’re angry, or afraid of the drain, or dealing with a divorce. The internet is desperately missing a place like the tagline of the documentary describes, a place that is “Deep and Simple” and can introduce these complicated topics in a manner that any child can approach.

There’s nothing technical missing. All the pieces, tools and technologies  are in place to build a world of exploration and acceptance online. But we’re waiting on our Fred Rogers. Someone who can come along and bring that touch of humanity and show us the way to use this new medium as a way of forging deeper connections between us.  Someone who can refocus technologies built for consumption on to creation and exploration. A place where children can learn to private work out their feelings through music, song, story and images.

If anyone who ends up reading this knows about a project like this, please let me know. I’d gladly donate some of my time, and any technical assistance I could lend. Like many of my peers I grew up with Mr Rogers neighborhood on the TV, and I’d be proud to help work on the “Neighborhood” for the next generation.

“Brogramming” and sexism, and alcohol, have been a pretty hot topic in tech the last few months. I didn’t really have much of an opinion about the subject, but a brief twitter exhange left me thinking.

Tech is a Young Field and It’s Heating Up

Professional fields get stereotypes associated with them. Everyone thinks of lawyers as stuffy, accounts as uptight. Tech used to be the province of the stereotypical ubergeek, but with the current growth and the emergence of Facebook and the mobile space, we’re getting a chance to redefine our industry. The question is, what kind of culture do we want to create?

Brogramming as the Symptom of a Deeper Problem

Testosterone fueled lingo. Women as window dressing. Working hard, boozing hard. We’ve seen this kind of thing before, in every portrayal, true, or exaggerated, of the high school football/jock/in crowd. The quarterback running the school, cheerleader on one arm and the entire universe at his feet. The cafeteria is a structured hierarchy, varsity football here at the head table, soccer there, rednecks here, nerds in the corner…

How many people in tech experienced the downsides of an “in crowd” at some point in their life? Why is this pattern starting to reemerge in what once was the refuge of the loners and the outsides? Whatever the original intend of the brogrammer meme, there have been enough examples of this kind of thing that it’s time to make a full-stop and reevaluate what’s funny.

Lighten Up, it’s all meant to be (ironic|funny|self aware)*

I understand the ironic intent of brogramming, I really do. I have no problem with occasionally poking fun at our own political correctness. I’ve laughed among friends at my share of sexist/off color jokes and told a few. What we’re losing sight of is, that as in programming, context is absolutely everything. Not everything that’s appropriate among a small group of friends is appropriate for a technical conference, nor a meetup, and certainly not at a company sponsored event. Nobody tells dead baby jokes to their mother. It’s the same age old speaker’s rule “know your audience” coupled with the world’s most effective machine at removing context: the internet.

Let’s Take Charge

One of the best things about tech is it is a young industry that moves and adapts fast. We build things, create products out of nothing more than digitally encoded thoughts.  Lets do a little better job of policing our appearance and the culture we’re creating. We don’t need the political correctness police, but lets get control of ourselves before “programmer” becomes synonymous with “douche bag”. Lets build an industry where strength of idea, and execution stay the primary currency regardless of gender. One that has room for those of us who rage in the codebase and the club, and those prefer a quiet night of WoW with friends. If we can do that we’ll not only be one of the pillars of the new economy but also an industry built around inclusion and excellence.

It seems to be fashionable of late to malign the value of a college education.  Tech pundits like Peter Thiel is offering bright students a pile of cash to either not go to school, or to drop out and start a company.  For some really bright people, this approach might make perfect sense, but to suggest it in general is incredibly shortsighted. It’s undeniable many, many people waste their opportunity when they go to college.  They go to the wrong schools, for the wrong reasons without putting thought into what could be a critical investment. Some, by poor instruction, disability, or simply lack of motivation, go without enough of an educational foundation to gain anything from college. The main thrust of most of the bulk of the counterarguments ultimately comes down to “It costs too much”, “You can learn all those things outside of college”, “Most people don’t work in the fields they get their degree in anyway” All true, all well documented, and all ultimately misdirection.

All these criticisms, in my opinion are aimed at implementation issues, rather than the idea of University learning itself. A coworker made this point best, and to paraphrase him, college, at its best, is academic boot-camp. I’d call it a short term sacrifice to make a long term gain. At its best, getting a college degree is a chance to dive deep into knowledge that the needs of today would never afford. It’s a chance to learn how to write with grace, and to construct well thought out arguments. Most importantly, it’s the one time in most peoples lives when they can dive deeply enough into learning that they can uncover ideas that will induce a paradigm shift.

So why doesn’t it work out that way? Everyone wants to blame one thing which, in my opinion, is why the discussion is all hand wringing and no progress. The truth is more layered. The cost of a college education is undeniably increasing quickly. Among American Institutions, you have increasing number of (often wealthy) foreign students attending. Then there’s the fact that lots of people treat college as a four year party, interrupted only briefly by classes and studying. Top notch researches aren’t always the best instructors, which puts a greater teaching burden on TA’s and graduate students. To top it off a college education is no longer a guaranteed ticket to employment, if it ever was. There are a host of other problems with how the higher education system currently functions.

So why defend the system? In the end, there are some things that just require a lot of time and immersion to learn.  It’s certainly possible to learn these things by oneself, but for most people carving out enough time to do so is hard. Our society is full of distractions, add on the mental load of career employment and most people don’t have the mental discipline left to push themselves through college level material requiring hours of study to master. Generally people’s skill in their career generally appreciate in value over time, post high school is the cheapest time to make an investment in the future. Wait too long, and most people won’t be able to afford the time when our the value of our  that would ultimately make them better employees, citizens, and human beings. Pragmatically, getting a degree immediately after high school means sacrificing time when most people’s skills and employment value are at their lowest.

On top of these pragmatic reasons College campuses are still a breeding ground for creativity.  They’re a place for people to collaborate  and rub elbows with other intelligent people without the time pressure of the workplace  Many successful startups and business have been created from connections created, and research done on the grounds of Universities.  Even Thiel’s Paypal had it’s roots on University Drive adjacent to Stanford. This unstructured collaboration and research often pays dividends in personal connections made, even if the student doesn’t end up working in their original field of study.

Higher education certainly doesn’t have to be that path for everyone.  There is a shortage of skilled tradesman and there are plenty of other paths including self education, and entrepreneurship for young people to develop into knowledgeable adults. None of these things changes the fact that for many people, a college degree is still a great investment. In short, lets not get hasty and throw away a great thing to resolve some implementation issues.

Sometimes I’ll come across something that I read, or hear, or see, that causes a permanent shift in my thinking. I was reminded of this the today when I was idly browsing my copy of Stranger in a Strange Land by Robert A Heinlein. The plot of this excellent piece of science fiction revolves around the reintroduction to Earth of a man raised by Martians. When I first read the book, in high school, I can clearly remember how shocking it was to put myself in the mind of someone experiencing money, religion, and other very common human concepts for the first time. While I certainly haven’t adopted all of the philosophies espoused in the book, these ideas definitely altered my thinking, I’d like to believe for the better. In college, it was breakthroughs in understanding of induction, recursion, and functional programming all radically altered the way I approach problem solving in ways that reach beyond my programming skill. I remember a similar effect the first time I really wrapped the concept of “focus” in the martial arts.

the common thread between all these personal mental shifts was a pile of hard work.  There’s currently no easy way to shortcut the learning process, and I can’t see any way to do so in the near future. This presents a real problem, considering that the amount of knowledge available to the average person is increasing exponentially. Most of us don’t have the luxury of having unlimited time for study and exploration. Technology might be make this information more accessible, but it isn’t making it any easier for our brains to process, collate, and learn from the increasing river of useful knowledge, nor does it make it easier to find which nuggets in the sea of content will actually spark a new idea.

I wonder if it would be possible to somehow speed up this process by collecting a list of books, activities, and conceptual “big ideas” that other people have found to create a paradigm shift in their own thinking. It seems to me that this would be a great way to continue working out my mental muscles without having to try to find  good ideas by luck or coincidence. I’d be much more motivated to study a subject outside my expertise  on my own if I knew what the “big idea” was that I was trying to wrap my head around.  Admittedly, this would come with the downside of knowledge “a mile wide, and an inch deep” but I think that’s a worthy sacrifice to prevent the motivation killing problem of drowning in the details of a new area of study.  The goal wouldn’t be to become a master of every domain, but rather, to force the mind to expand and grow by mastering new core ideas that change how one views the universe.

Does anyone have any suggestions for content that meets the standard of “paradigm shifting” ?  Or is it possible that this kind of thing is too personal to generalize well?

Programmers write code for a living.  Everyone who writes about programming agrees that programmers need to write more than just code, but this dramatically underestimates the level at which we as a profession should be writing.

It’s become a mantra to “show me the code” but the code isn’t always enough.  Code is a slippery and imperfect representation of the a particular programmers mental model spelled out in the language of the his or her  tools at a particular point in time.  This references changes meaning changes as languages and  frameworks changes, and the programmers working on it change.  Codebases live on, 2,5,10 years past their intended use.  We talk about producing literate code, testable code, readable code, but imagine code being written now viewed from the vantage point of a programmer 10 years from now.  I see it as most likely looking like COBOL appears to most programmers today.

This isn’t writing documentation for the codebase, comments inside of the code, or lists of features and technical specifications.  The truth of the matter is, useful though all those things may be, people don’t read them unless it’s absolutely unnecessary to solve a particular problem with your software. The kind of writing I’m talking about is crafting the story of what your software was intended to do, and to tell the origin of how that software was written.  It’s telling about the technologies used without diving into technical details by describing how they are being used to solve a particular problem.  In fact, a good litmus test to ask about whether or not something belongs in this type of document is “would a non programmer find this useful?”  If so, you’re probably on the right track, if not, keep writing it but put it in one of the aforementioned types of documentation.

Good writing isn’t boring, and it isn’t always entirely efficient.  That’s okay.  Developing a voice and keeping it light means someone might actually read your writing, rather than using ctrl-f to get to the point immediately.  Source control allows you to track the development of software over time, so too should the writing I’m describing document the changing goals and trials, both technical and business specific that the software is adapting to solve over time.  For a completely contrived, and short example:

As the load began increasing in our application and we reached a dozen servers we started having serious concerns about how to quickly get messages between the different parts of the application.  We opted to deploy the third party NServiceBus library as a quick and easy way to pass events between the different parts of our product.   This gave us a stable way to reliably queue up messages to ensure that the email system always sent out responses after the customer sent in an order and prevented embarrassing situations like last week when a silently failing service meant four people didn’t get their Friday deals email until Sunday morning…

This kind of documentation helps everyone.  It gives the business people an idea of how the application is working, it gives future programmers an idea of what problems were being solved when features were being written and it adds to the corpus of knowledge that sales, and management can use to better understand what the software team has been working on.  For consulting, this type of writing can quickly clear up unclear points that can otherwise get glossed over in discussion.  I’ve found that misunderstandings that continue for days during verbal discussion often get immediately highlighted when someone puts down their model of a situation into a problem.  Unlike more specific technical documentation, when paired with a time and a date this piece of writing never becomes outdated it’s just one more thing that paints a picture of how an application came to be what it is today.

The critical point in all of this is that only the people who are intimately involved in writing the application can write this kind of documentation.  Our frequent failure to do so is a contributor to engineering’s too often strained relationship to management and business, and it doesn’t need to be this way.  We can do better, and we can start doing so by communicating our work in a way that everyone else can understand and appreciate.  Write it up, stand up, and be recognized.