I had a slew of IIS FTP logs to dig through for a work project. I needed unique visitor details, and I needed it from a month of logs. Some of these daily logs were in excess of 40mb. I threw together a PowerShell script to do just so, kicked it off, and waited. And waited. I reconfigured the script to be more efficient in its processing, kicked it off again, and waited. And waited. And researched, because this was taking far too long. And – late to the game, I know – I found Microsoft’s Logparser. Logparser is a free tool, and it’s filled with magic. With one simple query it managed to pull the raw data I needed from the log files in *seconds*.
1 2 3 4 5 | #log files in c:\temp\iis, results to same path logparser "SELECT DISTINCT c-ip, cs-username INTO C:\temp\iis\results.txt FROM 'C:\temp\iis\*.log'" -i:W3C -o:W3C # |
In less than a minute I had a text file of unique visits gleaned from 550 MB worth of plaintext log files. From there I could use PowerShell to filter out any IP dupes (some connections do not have user details, thus producing some leftover duplication) and resolve hostnames from IP addresses to create a final masterlist. I’m not sure how Logparser does what it does so quickly.
