Labels

Thursday, May 15, 2008

PowerShell FindFirstFileW bypassing MAX_PATH

By default it seems PowerShell uses the ANSI versions of FindFirstFile and FindNextFile, and is therefore limited to MAX_PATH - 260 characters in total. This PowerShell script uses in-line compiled VB.Net to call the wide unicode versions - FindFirstFileW and FindNextFileW - to bypass the ANSI MAX_PATH limitations. The results are essentially the same as a 'dir /s/b/a-d' that won't return with 'The filename or extension is too long.' or 'The directory name x is too long' errors.

If you use '/l', the script will only return deep paths, and also provide the 8.3 equivalent to the deep path - a useful method to access these files. For example, using this method, you can access a file with UNC (eg \\server\share) over 18 levels deep ((260-16)/13).

Note that these wide calls only bypass MAX_PATH when using a mapped drive with the \\?\ prefix to disable path parsing. Just specify a mapped drive or local path normally, eg c:\temp, the script will automatically prepend \\?\

I've been experimenting with using PowerShell to dynamically compile VB.Net or C# code, and within that managed code, calling unmanaged platform invoke operations to get to APIs. I like the flexibility of using a scripting language rather than compiled code, and while this certainly isn't as functional as 'dir', it was useful to me when at least trying to get a list of deep files.



## FindFiles.ps1 ##
param(
   [string] $dirRoot = $pwd,
   [string] $Spec = "*.*",
   [bool] $longOnly = $false
   )

# Changes:
#  23/05/2008, Wayne Martin, Added the option to only report +max_path entries, and report the short path of those directories (which makes it easier to access them)
#
#
# Description:
#  Use the wide unicode versions to report a directory listing of all files, including those that exceed the MAX_PATH ANSI limitations
#
# Assumptions, this script works on the assumption that:
#  There's a console to write the output from the compiled VB.Net
#
# Author:
#  Wayne Martin, 15/05/2008
#
# Usage
#  PowerShell . .\FindFiles.ps1 -d c:\temp -s *.*
#
#  PowerShell . .\FindFiles.ps1 -d c:\temp
#
#  PowerShell . .\FindFiles.ps1 -d g: -l $true
#
# References:
#  http://msdn.microsoft.com/en-us/library/aa364418(VS.85).aspx
#  http://blogs.msdn.com/jaredpar/archive/2008/03/14/making-pinvoke-easy.aspx 

$provider = new-object Microsoft.VisualBasic.VBCodeProvider
$params = new-object System.CodeDom.Compiler.CompilerParameters
$params.GenerateInMemory = $True
$refs = "System.dll","Microsoft.VisualBasic.dll"
$params.ReferencedAssemblies.AddRange($refs)

$txtCode = @'
Imports System
Imports System.Runtime.InteropServices
Class FindFiles

Const ERROR_SUCCESS As Long = 0
Private Const MAX_PREFERRED_LENGTH As Long = -1

  _
Public Structure WIN32_FIND_DATAW
    '''DWORD->unsigned int
    Public dwFileAttributes As UInteger
    '''FILETIME->_FILETIME
    Public ftCreationTime As FILETIME
    '''FILETIME->_FILETIME
    Public ftLastAccessTime As FILETIME
    '''FILETIME->_FILETIME
    Public ftLastWriteTime As FILETIME
    '''DWORD->unsigned int
    Public nFileSizeHigh As UInteger
    '''DWORD->unsigned int
    Public nFileSizeLow As UInteger
    '''DWORD->unsigned int
    Public dwReserved0 As UInteger
    '''DWORD->unsigned int
    Public dwReserved1 As UInteger
    '''WCHAR[260]
      _
    Public cFileName As String
    '''WCHAR[14]
      _
    Public cAlternateFileName As String
End Structure

  _
Public Structure FILETIME
    '''DWORD->unsigned int
    Public dwLowDateTime As UInteger
    '''DWORD->unsigned int
    Public dwHighDateTime As UInteger
End Structure

Partial Public Class NativeMethods
   
    '''Return Type: HANDLE->void*
    '''lpFileName: LPCWSTR->WCHAR*
    '''lpFindFileData: LPWIN32_FIND_DATAW->_WIN32_FIND_DATAW*
      _
    Public Shared Function FindFirstFileW( ByVal lpFileName As String,  ByRef lpFindFileData As WIN32_FIND_DATAW) As System.IntPtr
    End Function
  
    '''Return Type: BOOL->int
    '''hFindFile: HANDLE->void*
    '''lpFindFileData: LPWIN32_FIND_DATAW->_WIN32_FIND_DATAW*
      _
    Public Shared Function FindNextFileW( ByVal hFindFile As System.IntPtr,  ByRef lpFindFileData As WIN32_FIND_DATAW) As  Boolean
    End Function

    '''Return Type: BOOL->int
    '''hFindFile: HANDLE->void*
      _
    Public Shared Function FindClose(ByVal hFindFile As System.IntPtr) As  Boolean
    End Function

    '''Return Type: DWORD->unsigned int
    '''lpszLongPath: LPCWSTR->WCHAR*
    '''lpszShortPath: LPWSTR->WCHAR*
    '''cchBuffer: DWORD->unsigned int
      _
    Public Shared Function GetShortPathNameW( ByVal lpszLongPath As String,  ByVal lpszShortPath As System.Text.StringBuilder, ByVal cchBuffer As UInteger) As UInteger
    End Function

End Class


Private Const FILE_ATTRIBUTE_DIRECTORY As Long = &H10
    Dim FFW as New NativeMethods

Function Main(ByVal dirRoot As String, ByVal sFileSpec As String, Byval longOnly As Boolean) As Long
    Dim result As Long

    result = FindFiles(dirRoot, sFileSpec, longOnly)

    main = result          ' Return the result
End Function

Function FindFiles(ByRef sDir As String, ByVal sFileSpec as String, Byval longOnly As Boolean) As Long
    Const MAX_PATH As Integer = 260
    Dim FindFileData as WIN32_FIND_DATAW
    Dim hFile As Long
    Dim sFullPath As String
    Dim sFullFile As String
    Dim length as UInteger
    Dim sShortPath As New System.Text.StringBuilder()


    sFullPath = "\\?\" & sDir

    'console.writeline(sFullPath & "\" & sFileSpec)

    hFile = FFW.FindFirstFileW(sFullPath & "\" & sFileSpec, FindFileData)     ' Find the first object
    if hFile > 0 Then            ' Has something been found?
      If (FindFileData.dwFileAttributes AND FILE_ATTRIBUTE_DIRECTORY)  <> FILE_ATTRIBUTE_DIRECTORY Then  ' Is this a file?
        sFullFile = sFullPath & "\" & FindFileData.cFileName
        If (longOnly AND sFullFile.Length >= MAX_PATH) Then
          length = FFW.GetShortPathNameW(sFullPath, sShortPath, sFullPath.Length) ' GEt the 8.3 path
          console.writeline(sFullFile & " " & sshortpath.ToString())  ' Yes, report the full path and filename
        ElseIf (NOT longOnly)
          console.writeline(sFullFile)
        End If
      End If

      While FFW.FindNextFileW(hFile, FindFileData)        ' For all the items in this directory
        If (FindFileData.dwFileAttributes AND FILE_ATTRIBUTE_DIRECTORY) <> FILE_ATTRIBUTE_DIRECTORY Then ' Is this a file?
          sFullFile = sFullPath & "\" & FindFileData.cFileName
          If (longOnly AND sFullFile.Length >= MAX_PATH) Then
            length = FFW.GetShortPathNameW(sFullPath, sShortPath, sFullPath.Length) ' GEt the 8.3 path
            console.writeline(sFullFile & " " & sshortpath.ToString())  ' Yes, report the full path and filename
          ElseIf (NOT longOnly)
            console.writeline(sFullFile)
          End If
        End If
      End While
      FFW.FindClose(hFile)           ' Close the handle
      FindFileData = Nothing
    End If

    hFile = FFW.FindFirstFileW(sFullPath & "\" & "*.*", FindFileData)      ' Repeat the process looking for sub-directories using *.*
    if hFile > 0 Then
      If (FindFileData.dwFileAttributes AND FILE_ATTRIBUTE_DIRECTORY) AND _
          (FindFileData.cFileName <> ".") AND (FindFileData.cFileName <> "..") Then
        Call FindFiles(sDir & "\" & FindFileData.cFileName, sFileSpec, longOnly)      ' Recurse
      End If

      While FFW.FindNextFileW(hFile, FindFileData)
        If (FindFileData.dwFileAttributes AND FILE_ATTRIBUTE_DIRECTORY) AND _
            (FindFileData.cFileName <> ".") AND (FindFileData.cFileName <> "..") Then
          Call FindFiles(sDir & "\" & FindFileData.cFileName, sFileSpec, longOnly)     ' Recurse
        End If
      End While
      FFW.FindClose(hFile)           ' Close the handle
      FindFileData = Nothing
    End If

End Function

end class

'@


$cr = $provider.CompileAssemblyFromSource($params, $txtCode)
if ($cr.Errors.Count) {
    $codeLines = $txtCode.Split("`n");
    foreach ($ce in $cr.Errors)
    {
        write-host "Error: $($codeLines[$($ce.Line - 1)])"
        write-host $ce
        #$ce out-default
    }
    Throw "INVALID DATA: Errors encountered while compiling code"
 }
$mAssembly = $cr.CompiledAssembly
$instance = $mAssembly.CreateInstance("FindFiles")

$result = $instance.main($dirRoot, $Spec, $longOnly)


Wayne's World of IT (WWoIT), Copyright 2008 Wayne Martin.

1 comment:

ewall said...

Hullo, Wayne-- Lately I've been using PowerShell to help cleanup permissions issues on a big file server, but am running into MAX_PATH restrictions. I was excited to find your post here and try it out, but sadly it's not working. The VB code is compiling as expected, it's just not looping or returning anything (if I uncomment the line to print the current path to the console, it outputs just once "\\?\F:\\*.*"). Any tips or suggestions?


All Posts

printQueue AD objects for 2003 ClusterVirtualCenter Physical to VirtualVirtual 2003 MSCS Cluster in ESX VI3
Finding duplicate DNS recordsCommand-line automation – Echo and macrosCommand-line automation – set
Command-line automation - errorlevels and ifCommand-line automation - find and findstrBuilding blocks of command-line automation - FOR
Useful PowerShell command-line operationsMSCS 2003 Cluster Virtual Server ComponentsServer-side process for simple file access
OpsMgr 2007 performance script - VMware datastores...Enumerating URLs in Internet ExplorerNTLM Trusts between 2003 and NT4
2003 Servers with Hibernation enabledReading Shortcuts with PowerShell and VBSModifying DLL Resources
Automatically mapping printersSimple string encryption with PowerShellUseful NTFS and security command-line operations
Useful Windows Printer command-line operationsUseful Windows MSCS Cluster command-line operation...Useful VMware ESX and VC command-line operations
Useful general command-line operationsUseful DNS, DHCP and WINS command-line operationsUseful Active Directory command-line operations
Useful command-linesCreating secedit templates with PowerShellFixing Permissions with NTFS intra-volume moves
Converting filetime with vbs and PowerShellDifference between bat and cmdReplica Domain for Authentication
Troubleshooting Windows PrintingRenaming a user account in ADOpsMgr 2007 Reports - Sorting, Filtering, Charting...
WMIC XSL CSV output formattingEnumerating File Server ResourcesWMIC Custom Alias and Format
AD site discoveryPassing Parameters between OpsMgr and SSRSAnalyzing Windows Kernel Dumps
Process list with command-line argumentsOpsMgr 2007 Customized Reporting - SQL QueriesPreventing accidental NTFS data moves
FSRM and NTFS Quotas in 2003 R2PowerShell Deleting NTFS Alternate Data StreamsNTFS links - reparse, symbolic, hard, junction
IE Warnings when files are executedPowerShell Low-level keyboard hookCross-forest authentication and GP processing
Deleting Invalid SMS 2003 Distribution PointsCross-forest authentication and site synchronizati...Determining AD attribute replication
AD Security vs Distribution GroupsTroubleshooting cross-forest trust secure channels...RIS cross-domain access
Large SMS Web Reports return Error 500Troubleshooting SMS 2003 MP and SLPRemotely determine physical memory
VMware SDK with PowershellSpinning Excel Pie ChartPoke-Info PowerShell script
Reading web content with PowerShellAutomated Cluster File Security and PurgingManaging printers at the command-line
File System Filters and minifiltersOpsMgr 2007 SSRS Reports using SQL 2005 XMLAccess Based Enumeration in 2003 and MSCS
Find VM snapshots in ESX/VCComparing MSCS/VMware/DFS File & PrintModifying Exchange mailbox permissions
Nested 'for /f' catch-allPowerShell FindFirstFileW bypassing MAX_PATHRunning PowerSell Scripts from ASP.Net
Binary <-> Hex String files with PowershellOpsMgr 2007 Current Performance InstancesImpersonating a user without passwords
Running a process in the secure winlogon desktopShadow an XP Terminal Services sessionFind where a user is logged on from
Active Directory _msdcs DNS zonesUnlocking XP/2003 without passwords2003 Cluster-enabled scheduled tasks
Purging aged files from the filesystemFinding customised ADM templates in ADDomain local security groups for cross-forest secu...
Account Management eventlog auditingVMware cluster/Virtual Center StatisticsRunning scheduled tasks as a non-administrator
Audit Windows 2003 print server usageActive Directory DiagnosticsViewing NTFS information with nfi and diskedit
Performance Tuning for 2003 File ServersChecking ESX/VC VMs for snapshotsShowing non-persistent devices in device manager
Implementing an MSCS 2003 server clusterFinding users on a subnetWMI filter for subnet filtered Group Policy
Testing DNS records for scavengingRefreshing Computer Account AD Group MembershipTesting Network Ports from Windows
Using Recovery Console with RISPAE Boot.ini Switch for DEP or 4GB+ memoryUsing 32-bit COM objects on x64 platforms
Active Directory Organizational Unit (OU) DesignTroubleshooting computer accounts in an Active Dir...260+ character MAX_PATH limitations in filenames
Create or modify a security template for NTFS perm...Find where a user is connecting from through WMISDDL syntax in secedit security templates

About Me

I’ve worked in IT for over 20 years, and I know just about enough to realise that I don’t know very much.