Agfdhyk

I need to scrape the html of a site, which is launched off a .url file, then find a certain line, and grab every line below it to a certain point. An example of the html code is below:

</p><ul><li>(None)</li></ul><h2><span style="font-size:18px;">Authorized Administrators and Users</span></h2><pre><b>Authorized Administrators:</b>

jim (you)

    password: (blank/none)

bob

    password: Littl3@birD

batman

    password: 3ndur4N(e&amp;home

dab

    password: captain



<b>Authorized Users:</b>

bag

crab

oliver

james

scott

john

apple

</pre><h2><span style="font-size:18px;">Competition Guidelines</span></h2>

I need to get all of the authorized administrators into a txt file, the authorized users into a txt file, and both into another txt file. Could this be accomplished with just batch and powershell?

^{I believe that this answer shows useful techniques, and I've verified that it works with the sample input, within the constraints stated. Do tell us (with words) if you disagree, so the answer can be improved.}

Generally, as stated, using a dedicated HTML parser is preferable, but given the easily identifiable enclosing tags in your input (assuming there'll be no variations), you can get away with a regex-based solution.

Here's a regex-based PSv4+ solution, but note that it relies on the input containing whitespace (line breaks, leading spaces) exactly as shown in your question:

# $html is assumed to contain the input HTML text (can be a full document).

$admins, $users = (

  # Split the HTML text into the sections of interest.

  $html -split

    'A.*<b>Authorized Administrators:</b>|<b>Authorized Users:</b>' `

    -ne '' `

    -replace '<.*'

).ForEach({

  # Extract admin lines and user lines each, as an array.

  , ($_ -split 'r?n' -ne '')

})



# Clean up the $admins array and transform the username-password pairs

# into custom objects with .username and .password properties.

$admins = $admins -split 's+password:s+' -ne ''

$i = 0;

$admins.ForEach({ 

  if ($i++ % 2 -eq 0) { $co = [pscustomobject] @{ username = $_; password = '' } } 

  else { $co.password = $_; $co } 

})



# Create custom objects with the same structure for the users.

$users = $users.ForEach({

  [pscustomobject] @{ username = $_; password = '' }

})



# Output to CSV files.

$admins | Export-Csv admins.csv

$users | Export-Csv users.csv

$admins + $users | Export-Csv all.csv

Assumptions are made about the desired output format (and HTML entities such as & aren't decoded), given that your question doesn't flesh out the requirements.

Here's my attempt to get what you are after.

$url        = '<THE URL TAKEN FROM THE .URL SHORTCUT FILE>'

$outputPath = '<THE PATH WHERE YOU WANT THE CSV FILES TO BE CREATED>'



# get the content of the web page

$html = (Invoke-WebRequest -Uri $url).Content



# load the assembly to de-entify the HTML content

Add-Type -AssemblyName System.Web

$html = [System.Web.HttpUtility]::HtmlDecode($html)



# get the Authorized Admins block

if ($html -match '(?s)<b>Authorized Administrators:</b>(.+)<b>') {

    $adminblock = $matches[1].Trim()

    # inside this text block, get the admin usernames and passwords

    $admins = @()

    $regex = [regex] '(?m)^(?<name>.+)s*password:s+(?<password>.+)'

    $match = $regex.Match($adminblock)

    while ($match.Success) {

        $admins += [PSCustomObject]@{

            'Name'     = $($match.Groups['name'].Value -replace '(you)', '').Trim()

            'Type'     = 'Admin'

            # comment out this next property if you don't want passwords in the output

            'Password' = $match.Groups['password'].Value.Trim()    

        }

        $match = $match.NextMatch()

    } 



} else {

    Write-Warning "Could not find 'Authorized Administrators' text block."

}



# get the Authorized Users block

if ($html -match '(?s)<b>Authorized Users:</b>(.+)</pre>') {

    $userblock = $matches[1].Trim()

    # inside this text block, get the authorized usernames

    $users = @()

    $regex = [regex] '(?m)(?<name>.+)'

    $match = $regex.Match($userblock)

    while ($match.Success) {

        $users += [PSCustomObject]@{

            'Name' = $match.Groups['name'].Value.Trim()

            'Type' = 'User'

        }

        $match = $match.NextMatch()

    } 

} else {

    Write-Warning "Could not find 'Authorized Users' text block."

}



# write the csv files

$admins | Export-Csv -Path $(Join-Path -Path $outputPath -ChildPath 'admins.csv') -NoTypeInformation -Force

$users | Export-Csv -Path $(Join-Path -Path $outputPath -ChildPath 'users.csv') -NoTypeInformation -Force

($admins + $users) | Export-Csv -Path $(Join-Path -Path $outputPath -ChildPath 'adminsandusers.csv') -NoTypeInformation -Force

When finished, you will have three CSV files:

admins.csv

Name   Type  Password      

----   ----  --------      

jim    Admin (blank/none)  

bob    Admin Littl3@birD   

batman Admin 3ndur4N(e&home

dab    Admin captain 

users.csv

Name   Type

----   ----

bag    User

crab   User

oliver User

james  User

scott  User

john   User

apple  User

adminsandusers.csv

Name   Type  Password      

----   ----  --------      

jim    Admin (blank/none)  

bob    Admin Littl3@birD   

batman Admin 3ndur4N(e&home

dab    Admin captain       

bag    User                

crab   User                

oliver User                

james  User                

scott  User                

john   User                

apple  User 

this is really rather ugly, and very emphatically fragile. a good HTML parser would be a better way to do this.

however, presuming you aint got the resources for that, here's one way to grab the data. if you REALLY want to generate two more files [Admin & User], you can do that from this object ...

# fake reading in a text file

#    in real life, use Get-Content

$InStuff = @'

</p><ul><li>(None)</li></ul><h2><span style="font-size:18px;">Authorized Administrators and Users</span></h2><pre><b>Authorized Administrators:</b>

jim (you)

    password: (blank/none)

bob

    password: Littl3@birD

batman

    password: 3ndur4N(e&amp;home

dab

    password: captain



<b>Authorized Users:</b>

bag

crab

oliver

james

scott

john

apple

</pre><h2><span style="font-size:18px;">Competition Guidelines</span></h2>

'@ -split [environment]::NewLine



$CleanedInStuff = $InStuff.

    Where({

        $_ -notmatch '^</' -and

        $_ -notmatch '^ ' -and

        $_

        })



$UserType = 'Administrator'

$UserInfo = foreach ($CIS_Item in $CleanedInStuff)

    {

    if ($CIS_Item.StartsWith('<b>'))

        {

        $UserType = 'User'

        continue

        }

    [PSCustomObject]@{

        Name = $CIS_Item.Trim()

        UserType = $UserType

        }

    }



# on screen

$UserInfo



# to CSV    

$UserInfo |

    Export-Csv -LiteralPath "$env:TEMPLandonBB.csv" -NoTypeInformation

on screen output ...

Name      UserType     

----      --------     

jim (you) Administrator

bob       Administrator

batman    Administrator

dab       Administrator

bag       User         

crab      User         

oliver    User         

james     User         

scott     User         

john      User         

apple     User

CSV file content ...

"Name","UserType"

"jim (you)","Administrator"

"bob","Administrator"

"batman","Administrator"

"dab","Administrator"

"bag","User"

"crab","User"

"oliver","User"

"james","User"

"scott","User"

"john","User"

"apple","User"