Reducing data footprints on test systems

Picture the scene:

You are a DBA. You have to make everyone happy, including DEVs (yes, including them too). Part of that is to ensure that your DEVs have a fresh copy of your production data so that they can test their programming with current data and structures. The issue with that is your production database is multiple hundreds of GB in size and your test system doesn’t have the capacity to store a full copy of your database. Your IT Budget will also not stretch to purchasing more storage! What can you do?

I had a similar situation recently and took a look at the possibilties and here is what I came up with:

1. Try out tools like SQL Virtual Restore to allow you to “mount” a backup from a remote server avoiding the space issue all together. This is, of course, a solution that requires a license (no budget maybe?)

2. Use the great features offered to you by SQL Server Developer Edition: data compression to the rescue!

So of course, I chose door number 2.

As you may know, Developer Edition offers the same features as Enterprise Edition with the caveat that the instance cannot be used for production purposes in any way. This means that the seriously powerful and useful data compression becomes available for your test servers! This counts even if you only use Standard Edition in production – joy of joys! The good thing being that Developer Edition only costs around $50 (or less if you have an MSDN subscription) – even more joys!

If you have never seen/used compression take a quick look over on MSDN to see what it all is (Data Compression). Basically, you can compress data in indexes and tables at the storage level avoiding any query rewrites and still profiting from the storage savings. This can be a major advantage in terms of raw storage needs, but can also benefit you in high read environments with low RAM sizes. The data remains compressed when held in the buffer pool and is only de-compressed when being accessed. This means that you can keep more data in the buffer pool and reduce hard disk I/O (obviously not for free, compressing and de-compressing costs CPU cycles).  This may be acceptable on a test system with extremely constrained storage space.

The usage in my example scenario is now an even better proposition, not only can I reduce my storage footprint, I can also potentially increase test system performance on the I/O side of the equation (who doesn’t have I/O issues, especially on a test box!).

The next hurdle is of course identifying which indexes and tables that are in the database you want to squeeze down. This is possible via SSMS’ object explorer, but only if you want to spend an eternity doing so! The best way is to take a look at the meta-data tables/views/functions to interrogate the system objects. These are really interesting on their own (at least they are to me), but after writing my Index Script Creator, I realised the potential for using the information in these system objects to allow me to automate certain tasks like this one.

Similarly to the Index Script Creator, my Index Compressor Script (uploaded as a doc  but it is a .sql file really – stupid wordpress restrictions!) runs off and investigates all tables and indexes (down to the partition level) and then generates an index rebuild command to compress any currently non-compressed partitions of indexes / tables. The resulting commands can then be run and space can be reclaimed – just be aware that rebuilding indexes does cause log activity.

After compressing your tables and indexes you should have a much smaller amount of data (depending on how well your data compresses) and be able to reclaim some disk space.

I realise that this method will mean that you have to have a large enough disk to have the database in its uncompressed state to begin with, but you can end up with a cost effective solution to a budget-constrained test environment.

Happy zipping!

SQLRoadtrip, part the 6th

My #sqlroadtrip is now over (part 1, part 2, part 3, part 4 and part 5). I have had 2 1/2 weeks of offline time, where I ignored technology as far as I could and spent some much needed time with my family. During that time I had two more flights and a hotel stay, although a much more relaxed version (a week in Sardinia and not a server in sight!).

@Fatherjack (Web|twitter) requested a DBA-typical numbers breakdown of my trip, I have attempted to oblige him here:

38 days
20 flights
15 countries
11 servers
11 hotels
5 continents
4 power cuts
1 case of food poisoning

In that time, I racked up 37229 miles (according to and reached the “Frequent Flyer” status with Lufthansa. I had a total flight time of approximately 80 hours (according to Lufthansa’s site), with the shortest flight being 150 miles and the longest 6526 miles.

I am happy to say, the rollout went well. The systems have been up an running since, and even survived a 5 day outage in one of the offices. The office came back online and synched up within a few hours, I was most surprised at that!

I am now easing myself back into my daily work, with an eye on my next trip to the PASS Summit in October – I hope to put my new Lufthansa status to use, maybe even upgrade this time round!

P.S. A final picture update is now online, with a few sites from Rio. Really nice city, if it wasn’t for all the smog!!

Be prepared!

While I was on my recent trans-global sojourn I came to the conclusion that the boy scouts really had a valid point with their motto “Be Prepared”. Of course, I am not meaning it quite in the semi-military sense that Robert Baden-Powell meant, but being prepared before the large system roll-out really saved a lot of time, nerves and money.

The roll-out was a set of reporting servers in a transactional replication setup, pushing operational data from a central server to the outlying branch offices around the world. I spent a great deal of time preparing these systems for the roll-out; standardising the installations, scripting out each step required to set up users, roles, permissions, DB objects, linked servers, jobs etc. This long preparation period was very tedious and I would often lose the motivation to keep at it.

The final payoff for this long drawn-out process has been the actual roll-out at each location. I have been able to arrive at each office, totally jet-lagged and tired from the entire travel regime and basically sit and watch the scripts run through and see the systems come to life. Had I not been prepared, I would have been in a world of pain, trying to remember what needed doing and when whilst fighting off sleep and headaches.

As a side note: A former colleague/mentor told me to save every script I ever run. If you don’t, you’ll need to repeat that script that took you an hour to write the very next day after you deleted it. This task has been made much easier to do thanks to the SSMS Toolspack provided by the more than awesome Mladen Prajdić. His SSMS extension has saved me hours of time when I have written code and accidentally thrown it away, or when SSMS has crashed. Use his tool and donate to the cause!

So, before you start any project, always keep the Scouts motto in mind – “Be prepared”. Hard disk space is so cheap as to be free, how cheap is your time?

The Home Stretch

I am now on the final third of my #sqlroadtrip (part 1, part 2, part 3 and part 4).

My last post saw me in New Zealand where I spent a great 36 hours taking in what I could of Auckland. Unfortunately I didn’t meet up with Dave Dustin, his work got in the way at short notice 😦 – this didn’t stop me from checking out quite a bit of Auckland, including the SkyTower. I got a great view of the city and then walked around town taking in the sights and sounds.

The evening flight from Auckland to Los Angeles ended up being really comfortable. Air New Zealand’s Premium Economy is a major step up from standard economy, offering miles of legroom and the business class menu and winelist (for a snip at $150 on top). I didn’t sleep, but was definitely comfortable for the 12 hour flight. This was followed by a 90 minute layover at LAX, where I was given priority entry into the US so that I could get to my connecting flight on time.

Calgary offered up another uneventful installation, allowing me and a colleague to check out the Banff National Park. It has some great views, with some of the mountain ranges looking like the mountains from the start of my trip in Switzerland (decide for yourself by checking out my web-album – link at the bottom of the page).

As luck would have it, when I checked into my flight from Calgary to Houston, I was offered an upgrade to Continental First Class for the wallet-friendly price of $100. I took them up on the offer and had a very comfortable 4 hour flight down to Houston. I was picked up from the airport by my data-courier colleague who had landed in Houston a couple of hours before me. He had managed to get a nice rental car upgrade because the rental company had messed up the reservation and only had a big SUV left! 🙂 We took off and setup the server in Houston and were treated to a relaxing evening at the local IT manager’s house. We were able to cool off in his pool and enjoyed a few ice-cold beers. This was the first real chance of relaxation since starting the trip all those weeks ago – thank you Volker! After missing the chance to meet Dave Dustin in New Zealand, I got another opportunity to meet an online acquaintance from the ASKSSC forumOleg Netchaev – for lunch in Houston. It was great to put a name to a face.

I then flew down to Veracruz for the next server installation. This was a little touch-and-go, because the hardware had been stuck in customs for a few weeks (it had been sent before I started my trip!). This was sorted out whilst I was in Houston and the server was brought online just in time to get things setup before my flight down to Mexico.

I arrived and finished setting up the server in Veracruz, no problems, everything ran like clockwork. The local IT manager then took us for a quick drive round the city, showing us a few sights and letting us get an idea of what the “real” Mexico is like. We ended up eating some mexican food (although I skipped the pigs brain in tortillas!) and heading down to the Aquarium. Whilst there, we were offered the opportunity to get into a glass box and feed the sharks in the big shark tank. We jumped at the opportunity and spent a fascinating 15 minutes in the water with an oceanologist who speacialises in sharks. That was definitely an experience I will never forget.

From Veracruz I set off to Rio de Janeiro (via Houston – where I am right now) to setup the 11th and final server. This will be a short hop, arriving Thursday and leaving on Sunday, but at this point I really don’t mind. The trip has been really interesting, but extremely challenging, both physically and mentally, and I’m looking forward to going home.

Luckily, I will be arriving in Germany next Monday and taking a 3 week break from work (and possibly technology). I should be able to recharge my batteries and get back to some sort of normal daily routine. My son has been asking if I’m in the planes that fly over our house since the first week of my trip. He seems to be looking forward to me coming home, I was worried he wasn’t bothered or had even forgotten about me 🙂

So, time to sit back and relax before the second longest flight of this trip. The President’s Club is comfortable and quiet and I’ll be fine for the 4.5 hours I have till boarding.


P.S. I have recently updated my web-album with the pictures taken since my last blog-post, with views from Calgary, Houston and Veracruz.

Sqlroadtrip – Half Way Point

I am now at the half-way stage of my round the world trip (see part 1, part 2 and part 3).

After part 3 I went on from Kuala Lumpur to Perth. This time I flew with Singapore Airlines – an extremely good airline as far as I am concerned, the entire experience was positive. I had a short hop from Kuala Lumpur to Sinpagore and then a 5 hours flight to Perth.

There was a mix-up at my company regarding the data transport, so I arrived a full 24 hours too early in Perth. I finally caught up on some of the missing sleep – I slept a full 18 hours!

The data finally arrived and I got to work at the Perth office. The installation ran without any issues (we’ve got the rollout totally sorted by now) and even managed to do a little more testing of the performance improvements of running readonly reporting on the local server. Who would have thought it, a server in the LAN is faster than a server over WAN!!! 😀

I only saw the hotel (Esplanade in Fremantle), the office and a micro-brewery pub (The Mad Monk in Fremantle). This was the shortest stop of all so far, but I was more than happy to take that hit to allow me to have some “real” free time in New Zealand.

I arrived in Auckland this morning after an uneventful flight from Perth to Auckland. I spent the day exploring the city and adding to the photo collection. I will hopefully be meeting up with Dave Dustin (Twitter) for lunch tomorrow before jumping on my longest flight so far – Auckland to Los Angeles to Calgary. The first leg is over 10000 km and 12 hours, luckily in Premium Economy flying with Air New Zealand – a little extra legroom is always nice.

So, time for bed, the day starts early with me trying to take part in the PoSh webcast held by Aaron Nelson (Web|twitter) for the UK Sout West User Group. That’ll mean a 5am start, but my brain is scattered from timezone-hopping anyway.

Round the world in 38 days – Teil 3

If you have been following my adventures so far, then you will know that I have embarked upon a trip around the world. This trip entails setting up a load of SQL Servers for my employer. For more details, see part one and part two.

I was unable to blog whilst in Dubai and Saudi Arabia, so I have a little catching up to do.

Dubai turned out to be the highlight of the trip so far. I was staying in the Mövenpick Hotel Ibn Battuta Gate. I managed to snag an extremely good deal on the hotel and upgraded to an executive room priced at 90 Euros a night. The hotel was absolutely brilliant and really proved the five star rating. The remainder of my trip will be comparing the other hotels to what was on offer in Dubai. I think it will be difficult to top though.

The roll-out has been pretty smooth so far. All those practice runs and copious notes seem to be paying off, who’d have thought!

Rolling into Saudi Arabia was a fun experience too. I landed at Dammam International Airport after a normal 90 minute flight from Dubai. The fun started at passport control, where I waited about 90 minutes to get processed. The men at the counters were rather lax in their work attitude, stopping to chat with their colleagues, smoke or just read a newspaper! I was luckily not waiting for a work-permit, some not so lucky Indians had already been waiting for 3 hours.

After finally collecting my suitcase, I went through to the arrivals lounge. It was then that I found out that no hotel and no pick-up had been arranged for me. Thus ensued a great deal of phoning around trying to find out who could help me out. As luck would have it, someone was still in the Dammam office. A car was sent to pick me up and a room was arranged for me at the Dhahran International Hotel. The hotel is a throw-back to the 70s (not a new hotel built to look that way). I’m sure I saw people with bell-bottom trousers and gold medaillions around their necks!

The roll-out in Dammam went equally smooth. The local IT man really looked after me while I was there. He took me to a couple of local restaurants for lunch, where I gorged on the brilliant arabian food on offer.

I then set off to my next destination – Kuala Lumpur. This was to be the longest leg of the trip so far, involving 3 flights. Dammam to Dubai, Dubai to Bangkok, Bangkok to Kuala Lumpur. I got to Dubai without any problems and treated myself to 3 hours of lounge access. I attacked the buffet and had a few beers to relax before the 6 hour flight to Bangkok.

The real fun started on the flight to Bangkok – Thai Airlines Flght TG 518. This will be burned in my brain until I die. I had a comfortable seat (32A) in a relatively new A330. The flight started nicely and I got stuck into the onboard film. Things changed when the food arrived. I tried the side salad (something with fish) and it didn’t taste right. 20 minutes later, my body decided that the fish really wasn’t right. I spent the remainder of the flight in a delirious cold sweat, running to the bathroom to throw-up. I can safely say, that was the longest 5 1/2 hours of my life.

The flight from Bangkok to Kuala Lumpur was a little better as I fell into a deep sleep.

All is well again this morning. I setup the system in Kuala Lumpur and have a day off tomorrow. I hope to climb the Patronas Towers and take a look around the city before flying to Perth on Sunday.

So, time to shut up shop for the day and relax a little. I’m only half way round the world and the travelling and time difference are certainly taking their toll!


Sqlroadtrip part deux

Hello again!

Here is an update on the #sqlroadtrip, which started out in Switzerland (part one of the trip).

I have spent the last few days in Kiev in the Ukraine. I arrived at 01:30 (EET) at the airport and spent the next 45 minutes waiting for passport control. They had 8 control gates, but only two open!? They are seriously expanding the airport in preparation for the european football championships next year. This is also the reason for all the road works in and around the city.

I was driven from the airport to the hotel (Rus Hotel, Kiev), where I slept until 9am (local time = CET+1). Then setup our replication system. After 4 hours, and only one error message, I was done!

I went to the hotel, had some rather uninspiring food and some cheap Danish and Ukrainian beer, then fell into a deep sleep.

The following day was spent checking the replication was working properly (it was) and explaining the replication system to the people in the office. We are running transactional replication to allow reporting offloading for each office. It is quite impressive to see our data being updated at near real-time (~5 seconds), even to the offices with the worst network connection. They were happy about the new system, but also gave me a lot of information to take back to HQ regarding performance problems. The perceived performance is different to the measured performance. I have already identified areas in our software that can be optimised for a better user experience, where the devs back home would think the behaviour/performance is acceptable.

I will finish up today and make sure the next location is ready for the replication, then check out some sights around the centre of Kiev. After that, it will be time to pack up and get ready for my next flight. I check out of the hotel tomorrow morning at 08:00 (EET) and take a flight from Kiev to Vienna 10:50 (EET) departure, arriving 11:55 (CET). This is followed by a 90 minute layover and then an Austrian Airlines flight to Dubai (13:30 CET depart, 21:10 GST arrival).

The colleagues from the Kiev office have been really friendly and helpful the entire time. The same can be said for the majority of the people I met. I can definitely recommend Kiev as a holiday location, or at least as a place on a european tour. Most young people speak English, which is good if you can’t speak/read Ukranian or Russian!

Check back again soon for the next part of my #sqlroadtrip, where you will see what I get up to in the Middle East.

Just as a teaser, I will be staying in a slightly better hotel in Dubai, details to follow when I have arrived 🙂