Get-ChildItem vs Dir in PowerShell

Batch as usual

Recently I was in the need of modifying a huge amount of files across a network share. After installing Serviio media streaming service I noticed that it will crash "randomly". After checking the logs it was clear that ffmpeg was crashing when trying to open subtitle files that were not in the encoding indicated by the Serviio console. I needed a quick way to update all subtitle files to a common encoding, so I decided to convert all files to UTF8.

Since I use a windows environment I went directly to PowerShell and wrote this.

 Get-ChildItem -path .\ -filter *.srt -file  | ForEach-Object { (get-content $_.FullName) | Out-File $_.FullName -encoding utf8 }

This worked almost right except for files with "[" or "]" (among other symbols) on the name or path. To solve it, just added the "-LiteralPath" switch to tell powershell not to consider any wildcards on the path name and just use it exactly as it is.

 Get-ChildItem -path .\ -filter *.srt -file  | ForEach-Object { (get-content -LiteralPath $_.FullName) | Out-File -LiteralPath $_.FullName -encoding utf8 }

And done! All subtitles files in my media server are now in UTF-8 and Serviio will work without crashing. 

Too slow?

Performance however was a concern, I noticed this was a bit slower that it should, considering a fast network, small size of the subtitle files (avg<100kb) and how simple the process is. This single-line script has only 2 parts:

(get the files)  then for each file (convert it)

I started to dig a little bit into Get-ChildItem I found that there have been complains about its performance for some time, but it is much better now than in previous versions. Anyway I tried a different way to do that same first part of getting the files and compared it against Get-ChildItem.

Using "cmd /c" executed "dir /s /b <pattern>" and did some tests, local and over the network. See the image below for an example measuring the search for .exe files in another drive.



Both over the network and locally, the "dir" version worked faster, of course it grabs less information than Get-ChildItem, which actually creates an object around the file returned.

For a final test I then changed the original script to: 

cmd /c dir /s /b *.srt  | foreach { (Get-Content -LiteralPath $_) | Out-File -LiteralPath $_ -Encoding UTF8 }

It works 100% like the original script and a bit faster. Although since the big chunk of  execution time belongs to the conversion of the files encoding, the jump in speed is not that big in this particular case. However, when I need to do search and filtering of files in the terms of thousands I no longer use Get-ChildItem.

Hope this is useful for you all.

Comments

Post a Comment

Popular posts from this blog

The case for a 4 week sprint

Image
Every time we talk about SCRUM, the length of the sprint is stated as a period of "two to four weeks", rarely, however, have I seen people going outside of the 2 weeks. This is not written on stone of course, and while 3 weeks might seems odd, there is a benefit to using a longer period for your sprints.  Image is taken from scrum.org Without any hard data to back that decision, the answer I get from asking about this decision seems to be related to "delivery speed", we assume that because we deliver value to the business every 2 weeks, we are in a more "agile" environment. Well, that might not be so true in every sense or for every project. Let's break some arguments: 1. Sprint length is not equal to the speed of delivery There is more to the speed of development and delivery than the timeframe of the delivery cycle, in a world with DevOps and full CI/CD implemented for your project, you could (should?) actually deliver features every single day all t...
Read more

NODE.js fs.readFile and the BOM marker

Working on a ReactJS project, got a small glitch while trying to read a JSON file and parse it as a javascript object. getDataObject() { var dataString = fs.readFileSync("./json/data.json", "utf8"); return JSON.parse(dataString); } output:   SyntaxError: Unexpected token   After checking, double and triple checking again that the file was correct I went deeper and realized that "fs.readFileSync" was returning me the BOM for the UTF file at the beginning of the string. So we just need to strip it out. getDataObject() { var dataString = fs.readFileSync("./json/data.json", "utf8"); return JSON.parse(dataString.replace(/^\uFEFF/, "")); } I then checked and there are "packages" for this, however I don't think is proper to just import a new dependency just to fix something this small. The big issue here is why is this considered correct. The BOM is not ...
Read more