Austin Agile DevOps

DevOps in the Cloud
posts - 69 , comments - 7 , trackbacks - 0

Tuesday, April 25, 2017

Redirect Standard Error to Output using PowerShell

We use LiquiBase for Database change control and Octopus for deployment to downstream environments in our CICD pipeline. Unfortunately, the version of LiquiBase we use writes information messages to standard error. Octopus then interprets this as an error and marks the deployment with warnings when in fact there were no warnings or errors. Newer versions of LiquiBase may have corrected this.

This statement in the update-database function of the liquibase.psm1 file will publish information level messages as errors in the Octopus task log:

..\.liquibase\liquibase.bat --username=$username --password=$password --defaultsFile=../.liquibase/ --changeLogFile=$changeLogFile $url --logLevel=$logLevel update

As a work-around, you can call the statement as a separate process and redirect standard error to standard out as follows:

&  cmd /c "..\.liquibase\liquibase.bat --username=$username --password=$password --defaultsFile=../.liquibase/ --changeLogFile=$changeLogFile $url --logLevel=$logLevel update 2>&1" | out-default

Now the messages are published to Octopus as standard output and display appropriately.

Posted On Tuesday, April 25, 2017 7:02 AM | Comments (0) |

Thursday, April 28, 2016

Installing Datadog on AWS Elastic Beanstalk Windows Instances

1.  Use ebextentions. Here’s an example.



2. Here’s the Powershell script. You’ll need to zip up the script and upload it to S3 per the ebextentions config file above.

$datetimef = get-date -format
$log = 'C:\ebextentions\datadog\install-datadog-agent-log.txt'

Write-Output ("$datetimef START install-datadog-agent.ps1") | Out-File $log -Append

$agentInstalled = sc.exe query | findstr DatadogAgent
if (!($agentInstalled)) {
	Write-Output ("$datetimef Datadog Agent service not detected, run installation") | Out-File $log -Append
    Write-Output ("$datetimef Source for install-datadog-agent.ps1 at DevOps.Aws\scriptsRun\utilities") | Out-File $log -Append
	$instanceId = Invoke-Restmethod -uri
	Write-Output ("$datetimef Assign InstanceId `"$instanceId`" as Datadog hostname") | Out-File $log -Append
	msiexec.exe /qn /i "C:\ebextentions\datadog\ddagent-cli.amd64.msi" APIKEY="######################" HOSTNAME="$instanceId" /Lp+ $log | Out-Null
    # check that agent service is now installed
    $agentInstalled = sc.exe query | findstr DatadogAgent
    if (!($agentInstalled)) {
    	Write-Output ("$datetimef Datadog Agent service not detected, run installation") | Out-File $log -Append
    	Write-Output ("$datetimef") | Out-File $log -Append
    	Throw ("$datetimef Datadog agent install failed") | Out-File $log -Append
    } else {
    	Write-Output ("$datetimef Datadog agent installed successfully!") | Out-File $log -Append
} else {
	Write-Output ("$datetimef Agent already exists, skipping install") | Out-File $log -Append

Write-Output ("$datetimef END install-datadog-agent.ps1") | Out-File $log -Append
Write-Output (" ") | Out-File $log -Append

Posted On Thursday, April 28, 2016 1:15 PM | Comments (0) |

Tuesday, February 16, 2016

TeamCity Build Number from Most Recent Git Tag + 1 Dash Short Sha

We want a semantic version build number derived from the most recent git tag that follows a three-node semantic version format.  We use GitLab and here’s how we do that in TeamCity.


1. Add a Powershell build step as follows:

$lastVerTag = git describe --abbrev=0 --tags --match [0-99][.][0-99][.][0-999]
[string]$majMin = $lastVerTag.Substring(0,$lastVerTag.LastIndexOf('.'))
[int]$incr = $lastVerTag.Substring($lastVerTag.LastIndexOf('.'))
$incr = $incr + 1
$semVer = $majMin + '.' + $incr
$Hash = "%build.vcs.number%"
$ShortHash = $Hash.substring(0,7)
Write-Host "##teamcity[buildNumber '$semVer-$ShortHash']"

2. In the Build Configuration Settings General settings set the Build numbe format field to: %build.number%


If the most recent Git tag using a valid three-node semantic version format was 0.0.0, then the build number would be: 0.0.1-fe46b10

Posted On Tuesday, February 16, 2016 12:42 PM | Comments (0) |

Tuesday, January 26, 2016

PowerShell Grant-EC2SecurityGroupIngress to All Traffic using AWS Source Security Group

I really had to dig for this, but it’s quite simple:

$natSG = Get-EC2SecurityGroup -Region $env:AWS_DEFAULT_REGION | ?{$_.Description -eq 'my NAT security group description' -and $_.VpcId -eq $vpcId }
$natSgGroupId = $natSG.GroupId
$defaultSG = Get-EC2SecurityGroup -Region $env:AWS_DEFAULT_REGION | ?{$_.Description -eq 'default VPC security group' -and $_.VpcId -eq $vpcId }
$sourceGroup = New-Object Amazon.EC2.Model.UserIdGroupPair
$sourceGroup.GroupId = $defaultSG.GroupId
$newIpRule = New-Object Amazon.EC2.Model.IpPermission -Property @{IpProtocol='-1'; FromPort='0'; ToPort='65535'; IpRanges=''; UserIdGroupPair=$sourceGroup}
Grant-EC2SecurityGroupIngress -GroupId $natSgGroupId -IpPermission $newIpRule

Posted On Tuesday, January 26, 2016 9:52 AM | Comments (0) |

Tuesday, December 2, 2014

Wanna use Firefox on the TFS web frontend?

TFS uses a version of NTLM over unsecure HTTP and Firefox doesn't want to allow this by default. Of course, Firefox doesn't tell you it's forbidding something; instead it just acts like you got a 403.

Here's how to fix it so you can use FF on TFS:

  1. In your address bar, go to "about:config"
  2. You should get a little search bar and a giant list of properties. In the search bar, type "ntlm"
  3. Look for the property "network.negotiate-auth.allow-insecure-ntlm-v1" and double-click the "false" in the Value column. It should boldface and turn to "true". The boldface indicates a user-set property.
  4. Restart FF and you're good to go.

Posted On Tuesday, December 2, 2014 11:55 AM | Comments (0) |

Tuesday, September 30, 2014

Microsoft.WITDataStore.dll not found: NuGet NuSpec References

I ported my TFS API code to  use the V12 (2013) assemblies. I then built and created a NuGet package of the TFS API Lib and installed it in an application. Immediately that application build failed because the Microsoft.WITDataStore.dll assmebly was missing from my TFS API NuGet package. 

I found that by default, when the reference tag is not used at all, NuGet adds all files included in the lib folder as references.  I needed the Microsoft.WITDataStore.dll in the package lib folder, but not added as as a reference (this fails).

The solution was to update my NuSpec file to explicitly mark the files I wanted added as references to the project.

<package xmlns="">
      <authors>Bob Hardister</authors>
      <description>A set of methods to get and set information in TFS 2013</description>
      <summary>Library of TFS API Methods.</summary>
        <dependency id="RavenDB.Client" version="2.5.2910" />
        <reference file="Microsoft.TeamFoundation.Build.Activities.dll" />
        <reference file="Microsoft.TeamFoundation.Build.Client.dll" />
        <reference file="Microsoft.TeamFoundation.Build.Common.dll" />
        <reference file="Microsoft.TeamFoundation.Build.Workflow.dll" />
        <reference file="Microsoft.TeamFoundation.Client.dll" />
        <reference file="Microsoft.TeamFoundation.Common.dll" />
        <reference file="Microsoft.TeamFoundation.VersionControl.Client.dll" />
        <reference file="Microsoft.TeamFoundation.VersionControl.Common.dll" />
        <reference file="Microsoft.TeamFoundation.WorkItemTracking.Client.dll" />
        <reference file="Microsoft.TeamFoundation.WorkItemTracking.Common.dll" />
        <reference file="Microsoft.TeamFoundation.WorkItemTracking.Proxy.dll" />
        <reference file="NLog.dll" />
        <reference file="TfsApiLib.Tool.Methods.dll" />
        <reference file="TfsApiLib.Tool.RunProcess.dll" />
      <file src="bin\$configuration$\Microsoft.TeamFoundation.*" target="lib\net45" />
      <file src="bin\$configuration$\Microsoft.WITDataStore.dll" target="lib\net45" />
      <file src="bin\$configuration$\NLog.dll" target="lib\net45\NLog.dll" />
      <file src="bin\$configuration$\TfsApiLib.Tool.*" target="lib\net45" />

Posted On Tuesday, September 30, 2014 8:41 AM | Comments (0) |

Friday, September 12, 2014

TFS Git Delete Remote Branch from Server

You can delete a published branch from Visual Studio, but to remove it from the server use the this git statement from the command prompt of the local repo:

C:\_s\Git\myRepo>git push origin --delete myBranch

Posted On Friday, September 12, 2014 6:36 PM | Comments (0) |

Friday, September 5, 2014

Team Explorer Everywhere Command Line Client on a Mac

I was surprised at the dearth of information on this. Here’s the whole enchilada. I did not have to enter user or password information for the TF commands. This could be because my Mac account and password is identical to the Windows AD account and password. Use TF help to get a full listing of commands and TF Help [command] for details on each command.

  1. Download TEE-CLC on your Mac and unzip/move the TEE-CLC-x.x.x folder under your Applications folder.
  2. Open the Terminal command line on you Mac (search for Terminal)
  3. To see the current execution path on your Mac:
    • echo $PATH
  4. Enter the following to add the TEE-CLC to you path:
    • export PATH=$PATH:/Applications/{TEE CLC folder name}
    • example: export PATH=$PATH:/Applications/TEE-CLC-11.0.0
  5. Enter the echo command again to verify the TEE CLC folder has been added to the path
  6. Install the Java JDK (installing just the run time is  not sufficient)
  7. You should be able to execute the TF command from the Terminal window once the Java JDK install has completed
  8. Create a folder to store your TFS source control files. I used \Documents\_s\MyCollection
  9. Accept the TCC license by executing:
    • tf eula
  10. Create the workspace:
    • tf workspace -new -collection:{url} -location:server {workspace name}
  11. Create the workspace mapping
    • tf  workfold -map -collection:{url} -workspace:{workspace name} '$/' '/Users/bob_hardister/Documents/_s/MyCollection'
  12. Navigate to your local workspace folder in the Terminal window to run TF Get and other workspace centric commands
  13. A Get command would then be
    • tf get -r  '$/…desried TFS server location to pull from'

Posted On Friday, September 5, 2014 11:58 AM | Comments (0) |

Friday, July 25, 2014

Move TFS 2013 Git Repo to a Different Team Project

1. Go to the target team project admin site

2. Create the new repo on the TFS server (with the same name as the source repo except under the target team project)

3. Go to the target team project site and display the empty repo. Note the new repo remote url

4. Open Visual Studio 2013 team explorer connect tab (Update 2 or >) and connect to the remote source repo

5. Clone the repo under the new team project or other location

6. Right click on the new (cloned) local repo and click Open Command Prompt

7. Copy the new remote url of the target team project repo (see #3 above)

8. Execute the following command in the command prompt:

[new local repo location]>git remote set-url origin [new remote url]

9. Right click on the new local repo in the Visual Studio 2013 team explorer connect tab

10. Click Open

11. Click "Unsynced Commits"

12. Click the "Sync" button to push the local repo to the remote

13. Go to the remote repo site and refresh to display the code just pushed from the local repo

Posted On Friday, July 25, 2014 12:34 PM | Comments (0) |

Tuesday, July 8, 2014

TFS 2013 Build Process Template – Change the “AutomatedTest” Argument Default Value

It should be simple, but due to the length of the string, the syntax was not obvious. The out-of-the-box default is as follows. Note the backslash escape character and stick to the syntax:

{New Microsoft.TeamFoundation.Build.Common.BuildParameter(" { ""AssemblyFileSpec"": ""**\\*test*.dll;**\\*test*.appx"", ""RunSettingsFileName"": null, ""TestCaseFilter"": """", ""RunSettingsForTestRun"": { ""ServerRunSettingsFile"": """", ""TypeRunSettings"": ""Default"", ""HasRunSettingsFile"": false }, ""HasRunSettingsFile"": false, ""HasTestCaseFilter"": false, ""ExecutionPlatform"": ""X86"", ""FailBuildOnFailure"": false, ""RunName"": """" } ")}

If you remove the values to expose the syntax you get:

{New Microsoft.TeamFoundation.Build.Common.BuildParameter("")}

To add another “test” to the template use this:

{New Microsoft.TeamFoundation.Build.Common.BuildParameter(""), 
New Microsoft.TeamFoundation.Build.Common.BuildParameter("")}

and just provide the desired values. Here’s what I ended up with:

{New Microsoft.TeamFoundation.Build.Common.BuildParameter(" { ""AssemblyFileSpec"": ""*UnitTests\\*Tests.dll"", ""RunSettingsFileName"": null, ""TestCaseFilter"": """", ""RunSettingsForTestRun"": { ""ServerRunSettingsFile"": """", ""TypeRunSettings"": ""Default"", ""HasRunSettingsFile"": false }, ""HasRunSettingsFile"": false, ""HasTestCaseFilter"": false, ""ExecutionPlatform"": ""X64"", ""FailBuildOnFailure"": false, ""RunName"": ""Unit"" } "), New Microsoft.TeamFoundation.Build.Common.BuildParameter(" { ""AssemblyFileSpec"": ""*IntegrationTests\\*Tests.dll"", ""RunSettingsFileName"": null, ""TestCaseFilter"": """", ""RunSettingsForTestRun"": { ""ServerRunSettingsFile"": """", ""TypeRunSettings"": ""Default"", ""HasRunSettingsFile"": false }, ""HasRunSettingsFile"": false, ""HasTestCaseFilter"": false, ""ExecutionPlatform"": ""X64"", ""FailBuildOnFailure"": false, ""RunName"": ""Integration"" } ")}

Posted On Tuesday, July 8, 2014 12:56 PM | Comments (0) |

Powered by: