Upgrading to Drupal Core 7.19

Drupal Core updates need manual intervention. It’s understandable that upgrading Drupal Core is more difficult than automatic upgrades of plugins. Having a running program upgrade it self while it is currently running is no easy task. I’ve held off upgrading the core on my web site for some time – my site is simple and the security patches haven’t affected us.

Unfortunately, my site admin has been diligently upgrading other modules over time. The drupal upgrade process for the most part has been painless. We were stuck in a position where the site required us to upgrade a dev module to a release module and we weren’t able to automatically. There was a strange dependency (that is currently slipping my mind) and we were forced to manually delete files and restart the entire server. Other than that I have been hands-off on the site for the most part.

Now unfortunately the latest Drupal Commerce depends on a newer version of Drupal Core. I’m surprised it let us put the system in an invalid state, but there’s no point complaining about or situation. The fix is simple enough – I need to upgrade Drupal Core!

For the most part I hope I just need to follow this: http://drupal.org/node/1494290

Part 1: Backup

I really should have a regular backup of the site. I don’t and for now I’m just interested in a one time backup. My site is hosted on CPanel and backing up a mysql database is shockingly simple. From the main dashboard you select Files->Backup->MySQL Backup and select your database. That downloads a SQL file. To restore a database, simply upload the downloaded file on the same page. Wow. That was too simple. There’s the Weekly Backup link staring me in the eye. I really should automate downloading those weekly backups.

Part 2: Install

The installation is simple. First, delete all files except for the Sites folder. Next download Drupal core from http://drupal.org/project/drupal and unzip everything EXCEPT for the sites folder. Login to Drupal and take the site out of maintenance mode. And finally restart the web server for the changes to take affect.

That’s it! So far everything is working great. Again Drupal has proven itself easy to manage – even for a clueless Drupal admin like myself. After logging in, I see all modules are up-to-date and hopefully there is no more maintenance for some time to come.

Posted in Drupal | Tagged | Leave a comment

Public Speaking & New Comp Sci Grads

I recently had the chance to talk to some computer science university students. I was there on behalf of my company with the intention of recruiting young minds away from sexier tech companies. I don’t do a lot of public speaking and in general I prefer to stay behind a computer screen and communicate digitally rather then face-to-face. Several of my colleagues have spoken at events like this on behalf of the company and it was my turn to do it. Like any task I must perform at work, I wrote a well thought through speech and delivered what I consider a compelling talk.

Public speaking is a rush! It’s also not nearly as scary as I remember it. I think the trick is to talk about something you care about and know your subject matter very well.

I talked about my career – this is something I care and know a great deal about. I shared the early days of my career, when I was first doing Co-op and how it contrasts to a Co-op student I recently hired. I made the argument that because my company is so amazing they will get great experience in X, Y and Z, whereas I was a mere code monkey at a seemingly great sweatshop. I hope it was inspiring. I figure even if they aren’t interested in joining my company I did tell them what they should be looking for their first position. I think it’s common sense, but it’s best to go somewhere where you get to learn. There’s the temptation to get any job, but the sooner they get a good job the sooner they’ll be heading in a direction best for them. Far too many companies are looking for code-monkeys to just crank out code. As a new grad that’s a horrible environment for learning. When starting out it’s important to work with smart developers that have time to mentor and train you. It’s also important for students to realize their education isn’t over. They MUST keep learning. You can certainly coast at your current education level for some time, but before you know it you’ll be wondering why your career has hit a ceiling and you are having a hard time finding that job you really want. Anyhow, maybe they’ll be more public speaking in the future. I’d like to do a tech talk on something… I’m sure I’ll figure something out.

Posted in Uncategorized | Tagged , | Leave a comment

SQL Server Auditing

I have a N-Tiered Web Application built on ASP.Net and SQL Server. The application needs to audit CRUD operations that take place on the Database capturing – Who did what and when. It needs to log what user, did what operation at what time and what changes occurred as a result of the operation. I’m exploring some of the newer features in SQL Server to see if they solve this problem. SQL Server has builtin Audit features – maybe those will meet my needs so this post will explore SQL Server Auditing.

Previous Work:

I’ve created two audit systems in the past. Both worked sufficiently well, but due to time pressures and personal ignorance, both suffered from differing levels of problems in the areas of performance, usability and maintainability.

First Attempt

The first system had a schema roughly like this. There was a Log table that stored the user id, action and timestamp of the audit event and a details table that stored all the properties that were modified in that event.

Log
| User | Action | Time Stamp |

Log Details
| Action | Property | Old Value | New Value |

This was implemented with a CLR trigger on CREATE, UPDATE and DELETE. It looped through the fields for a given insert or update and compared them to the previous values. The CLR trigger was more performant than trying to do this with a CURSOR. It was a simple architecture and quick to implement.

The downsides were that the CLR stored procedure was rather difficult to deploy and maintain – it was unlike the rest of our C# and SQL infrastructure. More importantly, the logged results were usually at too low of level to be well understood. There needed to be an additional business layer on top of the results to make sense of it and this layer was not very performant or useable. For maintenance, every new table needed the trigger to be added whereas doing logging in C#, can be done in a single base class with reflection.

Second Attempt:

The second attempt moved all the logic to the C# Business tier. Audit events are generated from the middle tier in the application. This is good in that the auditing is done at the business tier. The limitation is that if a change is directly made to the database it is not audited. The existing system has a few scenarios that require direct access to the database through Stored Procedures or direct SQL queries to get the required performance. Also the existing system involves storing temporal data changes for related entities such as Group Membership – Ie, if you want to find out ‘what did group X do in 2008’, you need to know who was in that group during that time period. There’s a fair bit of data needed to support this operation. If possible I would like to offload all of this data and logic to a separate Auditing System.

New System:

If possible I would like to offload the auditing logic and storage to a separate system. This is because my Dataset being audited is relatively small, but the number of audit events is relatively large given the update-frequency and that I require auditing of select statements as well.

The database I’m working with is relatively complex – probably 200-500 tables in the end. The tables represent 100+ entities within the system. Given this complexity a generic system is definitely preferred over manually logging each entity at the business tier. This generic system certainly can live at the business tier, but for now I’m exploring the feasibility of doing it at the database tier.

Arguably, an audit log using off-the-shelf SQL Server auditing functionality is more ‘trustworthy” or reliable than a roll-your-own solution. We develop at a high level of quality in general so I don’t see this as an overly strong motivator.

MSDN Resarch

To quite MSDN – “SQL Server audit lets you create server audits, which can contain server audit specifications for server level events, and database audit specifications for database level events. Audited events can be written to the event logs or to audit files. ”

I’m suspicious that this is only referring to system events such as “Rebuilding indices or Query Deadlocks” and not DML events. Further reading and this link makes me somewhat optimistic. It appears the “c2 audit mode” options records all queries in a rolling log file – perhaps similar functionality exists for inserts and updates.

So Auditing looks like it can be done. It can log all operations on a table to a binary file or the Windows Event Log. I have a hunch that this will log the raw SQL commands and not necessarily the before and after values of the data that was modified.

Well two unfortunate results:

 

1. SQL Server Enterprise edition is required to use ‘Granular’ Event Logging. Apparently ‘Basic’ logging is supported without enterprise edition, but I have not been able to figure out what Basic logging is. Luckily this post here has an example of the output.

Screen Shot 2013-02-09 at 3.52.36 PM

2. It only logs the SQL statements executed and not the affected data. This isn’t a big surprise, but it means SQL Server auditing will not meet my needs. Converting from arbitrary SQL statements to higher-level business entities is too complicated to be feasible.

Summary:

SQL Server auditing does not meet my needs. It looks best suited for applications where more logic is stored in the database. If an application used SQL users and stored procedures the builtin audit features would provide far more value. Next up I’ll look into SQL Change Tracking – I think it’s better suited for what I need.

Posted in Architecture | Tagged | 2 Comments

Building a site – Part 4

Version 1 Released!!!

It’s released! waidmonitor.com is live! It actually does something useful. Yay!  This is kinda cool. Now what? What do I want to do with it?

I’m not sure what to do with it. Honestly, towards the end I lost interest in it. It started as a way to learn and once I found the learning was no longer happening, my motivation to continue waned. In the future I think i’ll work towards projects that can be monetized – or some other carrot on the stick to help push myself towards the end goal.

It was super interesting. I learned a lot more than I had hoped for – especially on the design side of an application.

Lessons learned:

  • Start Small. VERY Small. It takes a surprisingly large amount of time to do anything. This is the single thing and I can’t repeat enough.
  • Really think hard about what you want to do before starting. I’m not talking about designing everything or developing a massive plan. Things take forever to build. It’s almost a certainty that the “I can build that in a weekend” app is actually a months worth of weekends.
  • Stay focused. Keep a plan, a list, notes, etc. Always know what to do next.
  • Show (the right) people. The right people can be incredible motivators. Showing another programmer a geeky tool like this can yield an excited response. Showing this to a girlfriend, regardless of how great, might not (and did not for me) yield a great response.
  • Enjoy it. If it starts to feel like work, you’re doing something wrong – change the plan, change what you’re doing, change something. It should be fun!

It was a fun project. Now I’m in the brainstorming stage of my next project… until next time. Mark.

Posted in .Net, Architecture, C# | Tagged , , | Leave a comment

Building a site – Part 3

Continuing Development as life happens

It is challenging to maintain momentum on a side project with Christmas, Snowboarding, work, TV and many other things going on. Those combined with a 3 week coding break almost killed my side project. Luckily I persevered and there’s something, albeit something very small, to show for it!

Even on a small project, there ends up being a fair bit of refactoring.  My initial attempt at DDD did not work well at all. It was very artificial and was difficult to integrate with the 3 distinct threads in the application. DDD is easily applied in an n-tiered stack with a single thread or on a web stack where it appears there’s a single thread, but on a moderately complex, multi-threaded environment it is very unnatural. Eventually I decided on a model with classes given appropriate component names. I also discovered a better way to detect what end users are doing – Global Event hooks is bad – GetLastInputInfo() is good. This was a core part of the application and combined with the DDD changes, the client code was almost completely re-done.

I feel as though I got lucky on the client side. My first attempt at the UI used Twitter Bootstrap  and that’s basically what you see today. It was a life saver really. I spent zero time on CSS, and only minimal time laying out the site using Bootstraps scaffolding. I definitely will use Bootstrap for future projects. If it was a professional project, I would need to invest time in customizing the CSS – since this is just for fun, the default looks great imo.

The project had several “very bad bugs”. For some time, the charts simply didn’t reflect what I had actually done! They kinda resembled what had happened, but it was easy to see they didn’t make sense – the application was completely useless even though it was correctly logging and displaying information. Somehow what you see was not correct. Debugging what was wrong with an ASP.NET MVC site hosted in Azure talking to Azure Table Service gave me considerable headache initially. After, I discovered that all it takes is a firewall rule in Azure and you can step through ASP.net code as usual, everything was simple.

Unit tests would have saved me incalculable amounts of time. I think all my really bad bugs would have been solved with proper testing. I had the attitude on this project that “I don’t need to write unit tests- thats’ a waste of time. My code is well-written. It could have unit tests written against it! – that’s good enough”.

Posted in .Net, Architecture, C# | Tagged , , | Leave a comment

Building a site – Part 2

Implementing towards Alpha

Now that I have the initial idea for waidmonitor.com, the next thing to do was to figure out the big technical unknowns of the project:

  1. How to get the current users window & detect if the user is moving the mouse or touching the keyboard.
  2. How to build a WPF application
  3. Azure.

I maintained a page (Chicken scratches is more realistic description) of the ‘vision’ of the project that I referred to keep me focused. At all times I maintained a very small prioritized TODO list of 5-8 items. I got one or two nights a week to work on this project so I would work on the top priority item. Note: Evernote is awesome for managing notes and links. I kept the list short and regularly threw out todo items if the list got too long or an item wasn’t as important as it was 2 weeks ago. Over time todo items naturally became evolved from epics ~ “Setup website on Azure” to “Add ‘That day has no history <div>’ with javascript when JSON contains no daily results”.

Ultimately I was simply applying several agile practices. They worked extremely well with 1-man project – more than anything they kept it structured and kept it moving forward.

Before long I had a prototype outputting information from Windows to a CSV file, a test using Azure Table Storage and a basic WPF application. Next I have to bring it all together…

Posted in .Net, Architecture, C# | Tagged , , , , , | Leave a comment

Building a site – Part 1

I built a small web application and I’m going to write about the design and thinking that went in to building it. This will be spread over several posts. If you want to skip all the reading the finished site is – http://waidmonitor.com.

1. Building a Site – Idea to Implementation

It all began with me asking myself ‘What am I doing?’ after a long day of work. The day had flown by and I had not completed anything I wanted to complete when the day began. I couldn’t understand where the last 8 hours had gone. I joked I should write an application that records what I do.

For some time I’ve become increasingly interested in Cloud computing and I’ve been looking for a small project that I could experiment with some cloud technologies. This seemed like a perfect fit – I would record my daily usage to the cloud, along with thousands of other users, perform never-before-attempted data mining, output reports and it would be a hit!

It started with me creating some mockups of the graphs I wanted to see. I knew I wanted something visual. I don’t want to be scanning through some spreadsheet of information. UI is definitely not my area of expertise – Steve Krugs “Don’t’ make me think” is the extent of my UI design skills. My initial sketches sure left a lot of unanswered questions, but it was a start.

Evernote%20Camera%20Roll%2020130107%20211245

Next I picked the core technologies.

  • ASP.NET MVC. This is what the charts would be displayed on. It’s an all round awesome framework and one I’ll use for just about any web project.
  • high charts (google’s top result for “javascript charts”)
  • WPF. This sounded more exciting than Web Forms. I’m picking the technologies so I’m going to pick fun ones.
  • Azure SQL Server. Sweet. SQL Server massively scalable (or so I thought)!

Obviously lots to do, but I now have the requirements, the tools I just need to build it!

Posted in .Net, Architecture, C# | Tagged , , , , , | Leave a comment