So, I’m bored and I’m tired of looking at an empty blog, time to share some of the stuff I’ve learned (and built) over the past few years.  A lot of it will be DBA based but I’ve always been a developer at heart so they are tools to help you do the job quicker.  I’m doing a lot of cooler database development work now so hopefully once I work though some backlog I’ll get to writing about more difficult things.

First, lets talk about a crawler briefly as most of the cool stuff I’ve done is based around data collection and storage.  I imagined a single data server being a central point for collecting data from all of the other data servers.  This works for most things in the database world, especially when you don’t have the luxury of pushing a service (application) to thousands of machines.  If you are into the service side, stay tuned, the team I just joined pushes data collection to a new level so hopefully I’ll get to shed some light on the concepts I’m learning.

Now, the central point without an application sitting on every machine can be achieved with openrowset because you can assume credentials between different SQL Servers on the same network (domain can of course be different).  This requires you to enable ad-hoc queries.  The way it is setup in SQL Server is beautiful because here you only have to enable openrowset on your crawler (central point of data collection) and you can leave the settings on the other thousand data servers alone.

There, now we’ve opened the door. The next step is to get the key (which you likely already have) to all the data servers.

Typically if you are managing even a couple hundred instances you will need a DBA functional group. This is a windows domain group where you can stuff all the DBA accounts so this way, when you build out a new server, you just add that group and everyone has access. We can the run the collection instance under one of those accounts and we have access everywhere (while obeying the domain account double-hop rule); OR we can create a common SQL authentication account everywhere and use that as the method to login everywhere. Additionally we can do some tricks via BAT files, but we will talk about that another time.

There you have it, now you’re primed and ready to go. Next post I’ll talk about the queries we can run to start collecting data.

Tags: crawler, openrowset, sql