Monday, November 19, 2018

Unable to GetPassword on AWS EC2 Launched from Windows Server 2016 Custom AMI

We found out that our custom AMI doesn't allow us to enable GetPassword from AWS console on any EC2 launched from it. After reading and some trial and error, we found out that InitializeInstance.ps1 has to be enabled for the next boot.

https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/ec2launch.html#ec2launch-inittasks

To be complete, LaunchConfig.json has to have adminPasswordtype  set to Random (default). Then, run the following PowerShell command:

C:\ProgramData\Amazon\EC2-Windows\Launch\Scripts\InitializeInstance.ps1 -Schedule

Thursday, November 1, 2018

Entity Framework (EF) Slow Insert and SqlBulkCopy to the Rescue

There was a need to insert bulk data into the database and we are using Entity Framework as our ORM strategy. However, bulk insert took forever. Disabling Auto Detect Changes helps a little. So I decided to look into SqlBulkCopy and ended up writing one for Entity Framework. It is generic enough and can be found in https://gist.github.com/nikyodo85/b82ffd56bb2f0d45a9860dadcdfdc01d.

It works well so far. Some of the drawbacks are:

  • It won't be able to auto insert relationship
  • No validation check
But very fast. To use it will be very similar to SqlBulkCopy:


Dim efSqlBulkCopy As New EFSqlBulkCopy(Of MyEntityClass)(myDbContext)
efSqlBulkCopy.WriteToServer(listOfMyEntityClass)

Supressing Output in PowerShell

I was working with DiskPart, PowerShell, and Amazon SSM. Whenever I run DiskPart from PowerShell, the output was reflected in the console and it was recorded as Amazon SSM run command output, thus my run command was not completely clean. Such as the following:

Microsoft DiskPart version 6.3.9600
Copyright (C) 1999-2013 Microsoft Corporation.
On computer: MyComputer
DISKPART>
Disk 1 is now the selected disk.
DISKPART>
DiskPart successfully converted the selected disk to dynamic format.

Reading Q&A and documentation online, seems like no straightforward way to suppress output from DiskPart. I then tested something, how about if I assign the output to a variable and ignore it. It works and the command is still executed. For example, listing volume will be like the following in PowerShell:

$DiskPartOutput = 'list volume' | diskpart

Thursday, October 25, 2018

From AWS System Manager Document to Linux EC2 PowerShell Script

Let me start from what I'm trying to accomplished. Basically, I need to map partitions in Linux with EBS volumes. I first tried "df" command which works great. Through the following article, I found a way to retrieve corresponding volumes attached to the EC2 instance.

https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/ec2-windows-volumes.html

Next, it is just a matter to marry both of them and output them as json. After many web searches and pages, I decided to install PowerShell Core to utilize its ConvertTo-Json function. All went great and I would like to create a new AWS System Manager Command Document so I can fire up an api call to execute the script.

I first tried AWS System Manager's RunPowerShellScript document which claims to be able to execute PowerShell script on Linux. However, it somehow failed. I checked the requirements, update SSM Agent to no avail. Eventually, we submitted support ticket and waiting for the reply.

While waiting for the support reply, I found a workaround. The RunShellScript document works fine and PowerShell supports running command as argument, so I tried:

pwsh -Command "<the_command>"

And it failed. Through trial and error, I found out that I need to escape the $ sign on the document side. I then tried again. And of course, it failed again.

Next, I remembered that double quotes needs to be escaped in the document. However, it can create confusion on the execution side, so it has to be escaped the second time for PowerShell argument.

On the document, it becomes:

" pwsh -Command \"\$PS_Variable=\\\"Intended String\\\"\""

The double escape works for the double quotes, but it is troublesome, so I decided to change the internal double quotes to single quote because PowerShell can handle it too and no escape needed.

" pwsh -Command \"\$PS_Variable='Intended String (cleaner, no?)'\""

However, I found out another issue, single quote does not handle insert of variable and thus for my case, it still failed.

" pwsh -Command \"\$PS_Variable='This does work'\""

" pwsh -Command \"\$PS_Variable='NOT' \$YetAnotherVariable='This does \$PS_Variable work'\""

Well, it is easy, just return it to double-escaped double quotes.

Next, I had another issue that it is unable to access AWS API. After another set of trials and errors, I found out that I need to import AWS Powershell module in the document, thus I have to add the following line:

" Import-Module AWSPowerShell.NetCore"

or shorthand:

" ipmo AWSPowerShell.NetCore"

And now it is 5 and works perfectly.

Edit 10/26/2018: AWS support replied that there is indeed an issue with the RunPowerShell script command.

Edit 11/01/2018: AWS support said the internal team were able to replicate the issue and working on the fix. No ETA as of today.

PowerShell Split String by Whitespaces or Multiple Spaces Using Regex

I was looking into a way to split string that can take regular expression in PowerShell. Surprisingly, they have the capability:

$StringArray = $StringInput -split '\s+'

Invoke-WebRequest PowerShell Command through Amazon AWS System Manager

I had a small issue with running Invoke-WebRequest through Amazon AWS System Manager. Somehow it doesn't seem to load the module properly.

I ended up replacing:

Invoke-WebRequest -Uri <url>

with:

$WebClient = New-Object System.Net.WebClient
$WebClient.DownloadString(<uri>)

which works perfectly for my case.

Wednesday, October 10, 2018

UNC and URL

Found something weird today. For a long time, we have published our project to the test environment using the following UNC pattern: \\testservername.productiondomain.com\shared-path. However, when I did that today, it accidentally connected to production.

Removing the domain as part of the UNC solves the problem. The weird thing is the same UNC points to our test environment when run on my coworker's computer. Nothing is weird on our hosts file.

I have not figured out why this is so. This post thus will be updated once I found out the actual reason.

Tuesday, October 2, 2018

Entity Framework Sorting/ Ordering and Dynamic LINQ

One of my coworkers would like me to update my IEnumerable SortBy extension method to support multiple columns. As I was working on it, I stumbled upon a nice StackOverflow question on Dynamic LINQ OrderBy on IEnumerable<T> and also Dynamic LINQ.

They are all awesome. I decided to try Dynamic LINQ by adding it through Nuget. However, I soon found that Dynamic LINQ only support property names. In some cases, we want to be able to sort by specifying column attribute value, so I still use my own code which works just fine although it might not be optimized for performance.

For example in VB.NET:

Public Class Customer
    <Column c_name="">
    Public Property Name As String

    <Column c_age="">
    Public Property Age As Integer
End Public

Dim sortedByName = dbContext.Customers.AsEnumerable().SortBy("c_Name, c_Age DESC")

The gist for my code can be found in:
https://gist.github.com/nikyodo85/202fc2d417d9eb030d30896ccc862b7d

Thursday, September 13, 2018

Entity Framework (EF) Decimal Mapping to SQL Server

We bumped into a strange issue. Value of our property was not saved to the database. We checked mapping, spelling, data type and none seems wrong.

Eventually we found out that the decimal in our database has precision of (12, 6) and the SQL default is (18, 2). Problem is EF map Decimal data type to the SQL default precision thus our value was truncated.

To fix the issue, we put the following code in our DbContext under OnModelCreating method and our value was then saved correctly.

protected override void OnModelCreating(System.Data.Entity.DbModelBuilder modelBuilder)
{
       modelBuilder.Entity<Class>().Property(object => object.property).HasPrecision(12, 6);
}

Autocomplete Bug in Chrome Form Field

I just update my website and everything works great in Firefox. As I test it in Chrome, I noticed when I navigate to another and then press the back button, my text field was populated with wrong data. The data was the default value of other text field before the navigation.

As I was looking around, I found out that it is a bug in WebKit based browsers. Some people suggested to use autocomplete="off" property on all input field. But for my case, it is enough to put the attribute to all text fields (<input type="text"/>).

Saturday, September 8, 2018

A2 Windows Hosting HTTPS Redirect Conflict

With the intention of following best practice, I tried to enforce HTTPS on my websites hosted in A2 Hosting. Under Hosting Settings > Security, I found a check box that says "Permanent SEO-safe 301 redirect from HTTP to HTTPS", it sounds like the right one as it will redirect any HTTP to HTTPS. Because it is a windows server, it also has IIS Settings. And I found "Require SSL/TLS" check box under IIS Settings > Directory Security Settings. That sounds different from permanent redirect, so I check it as well.

When I test the settings by visiting my website using regular HTTP, I received a 403 error. My first attempt to fix the issue was going back to the IIS Settings and set the Authentication from Windows to None. However  that doesn't work. After several trial and error, I have to uncheck the "Require SSL/TLS" check box under IIS Settings > Directory Security Settings to redirect the traffic correctly.

Wednesday, September 5, 2018

A2 Hosting with .NET Core 2.1

Technology advances so fast and I have a new web application which I built using .NET Core 2.1. However, my current web hosting provider does not support .NET Core and I have to look for a new web hosting.

Reading some forums, found out that A2 Hosting plans to install .NET Core 2.1 in their shared hosting on August 2018. So on September 5th, 2018, I signed up for a new account with them. The coupon HOSTINGFACTS that I found from a web hosting review website still works and gave me 53% one-time discount.

Everything went smoothly until I found out that in my Plesk, I was unable to go higher than .NET 4.6.2. So, I decided to chat with their customer support. It took a while to get in touch with an agent, which makes sense because they have lots of customers and it was during busy hour.

From the chat with their customer support, I found out that .NET Core 2.1 has indeed been installed and what is weird is I don't have to flip the .NET version in Plesk. With that information, I decided to try deploy my web application using web deploy mechanism. I had to enable this on the domain level and Plesk will add a link to download the .publishsettings file which can then be imported to Visual Studio.

Deploying my application using web deploy mechanism has a small issue too. A2 Hosting apparently uses self-signed certificate and Visual Studio rejects the deployment because it can't trust it. To workaround it, I found many solutions. One of them is to trust the certificate. However, I don't think it is a good idea to trust untrusted certificate, so I decided to add <AllowUntrustedCertificate>True</AllowUntrustedCertificate> tag on the publish profile xml file.

To add the tag, I can't find the UI in VS2017, so I have to go to my project folder, under Properties > PublishProfiles and manually add it to the .pubxml file.

At last I managed to deploy my application and it worked great. Check it out at: https://www.kodeseek.com :)

Wednesday, August 29, 2018

Entity Framework (EF) Include() Lambda Extension Method Namespace

As of this writing, seems like Visual Studio still unable to provide suggestion on what namespace to import for extension methods.

I was looking into doing eager loading in EF and I am aware that I can use Include() method with lambda function. By default, it is not available and I can't remember which namespace it is located under. Of course, Visual Studio was not much help. After searching online, I found out that it is an extension method under System.Data.Entity namespace. I gotta remember from now on.


Monday, August 27, 2018

Entity Framework (EF) 6 Database.SqlQuery Mapping

I bumped into a problem with mapping results from manually crafted sql query to object using EF. Somehow, it doesn't seem to recognize the column attribute. And I found out that it really doesn't and they don't plan to enhance EF 6, although there seems to have the option in EF Core to do the mapping.

To illustrate my problem better, suppose my database column name is pk and would like to map it to ID property. Usually, using [Column("pk")] will solve the mapping, however, it doesn't work if the query was executed via dbContext.Database.SqlQuery("SELECT pk FROM TableName").

So I have to either change the ID property to pk or the one that I preferred is to change the query to dbContext.Database.SqlQuery("SELECT pk AS ID FROM TableName").

Thursday, August 23, 2018

How to Craft Multipart Form Data?

Ok, it is all started from me trying to send image from android to my web service using Google Volley. By default, Volley doesn't have built-in request to send image, but allows you to extend the request class. So, I need to extend the request class and create a MultipartFormDataRequest kind of class.

Reading after reading, I can't find the information I need to craft one. So I have to combine whatever I read with trial and error. The first one is the concept of boundary. It is a required information under Content-Type header. It acts as a separator between fields and can be any random string as long as it meets the requirements such as it can't exist in actual field data. More information such as length and size limit can be found under RFC 2046 Section 5.1.

So, the content-type header will be:

Content-Type: multipart/form-data; boundary=anyrandomstring

Next is how to use the boundary. The http body will start with two hyphens followed by the boundary. Underneath it is the Content-Disposition and Content-Type key value pair for each field. Between each field, it will be another two hyphens followed by the boundary as the separator. The one that caught me off guard was the closing boundary, it is two hyphens followed by the boundary and then followed by another two hyphens, so the http body will be:

--anyrandomstring

Content-Disposition: form-data; name=textFieldName
Content-Type: text/plain

TextFieldValue

--anyrandomstring

Content-Disposition: form-data; name=imageFieldName; filename=imageFilename.jpg
Content-Type: image/jpeg

[ImageBytes]

--anyrandomstring--


And then it works wonderfully. :)