Deepu Mohan Puthrote

Clojure CDK

Sat, 09 Sep 2023 21:44:52 GMT

I have be been wondering how to do intrastructure as code in Clojure. So I made an example for it using AWS CDK.

You can find the example in this repo https://github.com/WarFox/clojure-cdk-example

The example can deploy an SNS Topic, SQS Queue and subscription. There is also a basic test setup in the example. One of the benefits of using CDK is that we get define tests for our infrastructure in the chosen language.

The hardest part in using CDK with Clojure is defining the dependency betweeen the app, stacks and resources. In Java it naturally managed by class inheritance and composition; an app is composed by stacks and stacks are composed by resources.

In Clojure with Java Interop, we can manage such dependency using something like Integrant. Integrant lets you define dependency between componets of your system using a simlpe data structure.

The following is the integrant configuration used in the example app.

(def config
  {:app/instance   {}
   :stacks/topic   {:app      (ig/ref :app/instance)
                    :stack-id "TopicStack"}
   :stacks/storage {:app      (ig/ref :app/instance)
                    :stack-id "StorageStack"}
   ;; make sure to add all stacks to :stacks key
   :app/synth      {:app    (ig/ref :app/instance)
                    :stacks [(ig/ref :stacks/topic)
                             (ig/ref :stacks/storage)]}})

Integrant will take care of initiating the components based on the configuration pass the components for synth.

To make the setup work with CDK, we also need to configure it in cdk.json file

{
  "app": "clojure -M:synth",
  ...
}

I also have the following alias is deps.edn to make it work.

:aliases
{;; clojure -M:synth
 :synth
 {:main-opts ["-m" "core"]}
  ...
}

Checkout the example here and give it a star if you like https://github.com/WarFox/clojure-cdk-example.

You can run cdk synth and cdk deploy after checkout.

I am not stongly recommending you to use Clojure with CDK. Probably Typescript is the best language natively supported by the CDK team. However, if you like to leverage Clojure's simplicity in CDK then go ahead give it a try.

There are several other ways to manage infrastructure using Clojure. MarketSplash has a good article about Clojure's role in devops. This has many examples for using Clojure for your infrastructure needs.

Here is another article about using Clojure with CDK https://matthewdowney.github.io/aws-cdk-clojure-cloudformation.html

Hope this example gives a reference for your next CDK project!

Sieve of Eratosthenes Algorithm

Thu, 22 Jun 2023 22:18:13 GMT

The Sieve of Eratosthenes is an algorithm for finding all prime numbers up to a given limit. It works by iteratively marking the multiples of each prime as composite (i.e., not prime), starting with the multiples of 2.

Here is how the algorithm works:

Create a list of all integers from 2 to n, where n is the limit of the search.
Start with the first prime number, which is 2. Mark all multiples of 2 as composite (i.e., not prime ).
Find the next number in the list that is not marked as composite. This number must be prime. Mark all multiples of this number as composite.
Repeat step 3 until all numbers up to n have been processed.
The unmarked numbers are all prime.

Here's how the Sieve algorithm can be implemented using lazy sequences in Clojure:

(defn sieve
  [s] ;; accepts a lazy sequence
  (let [p (first s)]
    (lazy-seq
      (cons p ;; start with p followed by result of recursively calling sieve
            (sieve (filter #(not (zero? (mod % p)))
                             (rest s)))))))

(def primes (sieve (iterate inc 2)))

(take 10 primes) ;; (2 3 5 7 11 13 17 19 23 29)

The sieve function takes a lazy sequence s and returns a lazy sequence of prime numbers. The first line of the function extracts the first element of s, which must be prime, and assigns it to the variable p. Then we create a lazy sequence that starts with p, followed by the prime numbers generated by recursively calling sieve with a filtered version of s that removes all multiples of p.

We are not explicitly marking numbers as composite, instead we are filter them out using mod function.

The primes variable is initialized to the lazy sequence of prime numbers generated by calling sieve with a lazy sequence of all integers starting from 2. Because the sequence is lazy, only the first few primes are computed and stored in memory at any given time.

And thus you have an infinite sequences of prime numbers at your hand.

Lazy Sequence in Clojure

Tue, 04 Apr 2023 11:38:58 GMT

Introduction

A lazy sequence is a sequence that is not realized (or computed) until it is needed. This can be useful for handling potentially infinite sequences or for avoiding unnecessary computation when dealing with large sequences. You benefit by avoiding computations until they are absolutely needed and on demand.

Lazy sequences are available directly in many functional programming languages like Haskell (LazyEvaluation),Swift (LazySequence) and via libraries in other languages like Rust (lazy_seq), Go (seq). We will look into lazy-seq available in clojure.core.

To create a lazy sequence, in Clojure you can use the lazy-seq function. Here's an example of how to create a lazy sequence of integers:

(defn number-sequence
  ([] (number-sequence 1))
  ([n]
   (cons n
        (lazy-seq
          (map inc (number-sequence n))))))

  (take 5 (number-sequence)) ;; (1 2 3 4 5)
  (take 5 (number-sequence 10)) ;; (10 12 13 14)

In the above example, number-sequence is defined as a function that returns a lazy sequence of integers starting at n. The lazy-seq function creates the lazy sequence, and the cons function is used to add the first element (1) to the sequence. The map function is used to generate the remaining elements of the sequence by adding 1 to each element of the previous sequence.

When we call (number-sequence), we get back a lazy sequence object. When we call take to get the first 5 elements of the sequence, Clojure will realize (or compute) only the first 5 elements of the sequence, rather than computing the entire sequence up front. This can be useful for avoiding unnecessary computation when dealing with large or potentially infinite sequences.

Before going to more detailed explanations, here are two more examples of how to use lazy-seq in Clojure:

Example 1: Generating an infinite sequence of Fibonacci numbers

(def fib-seq
  (cons 1
        (lazy-seq (cons 1 (map + fib-seq (rest fib-seq))))))

(take 10 fib-seq)

In this example, fib-seq is defined as a lazy sequence of Fibonacci numbers. The cons function is used to add the first two elements (1 and 1) to the sequence. The map function is used to generate the remaining elements of the sequence by adding each element to its predecessor. Because fib-seq is defined recursively in terms of itself, it creates an infinite sequence of Fibonacci numbers.

When we call (take 10 fib-seq), Clojure will realize only the first 10 elements of the sequence. Because the sequence is defined lazily, we can generate an infinite sequence of Fibonacci numbers without running into issues with stack overflow or memory usage.

Here is another way to create infinite fibonacci sequence in Clojure using lazy-seq

(defn fib
  ([]
   (fib 1 1))
  ([n m]
   (cons n (lazy-seq (fib m (+ n m))))))

(take 10 (fib))

Example 2: Implementing the Sieve of Eratosthenes algorithm

(defn sieve [s]
  (lazy-seq
    (let [p (first s)]
      (cons p
            (sieve (filter #(not= 0 (mod % p)) (rest s)))))))

(def primes
  (cons 2 (sieve (iterate #(+ 2 %) 3))))

(take 20 primes)

In this example, sieve is defined as a function that implements the Sieve of Eratosthenes algorithm, which generates a sequence of prime numbers. The lazy-seq function is used to create the sequence of prime numbers. The let expression is used to bind the first element of the input sequence s to the variable p. The cons function is used to add p to the output sequence. The filter function is used to remove all elements from s that are divisible by p. The resulting sequence is passed recursively to sieve.

When we call (take 20 (sieve (iterate inc 2))), we get back a lazy sequence of the first 20 prime numbers. Because the Sieve of Eratosthenes algorithm can be used to generate an infinite sequence of prime numbers, we can use lazy-seq to generate prime numbers on-the-fly as we need them, without having to compute the entire sequence up front.

How does lazy-seq work internally?

Let us get deeper in it.

When you call (lazy-seq <body>), Clojure returns a lazy sequence object that wraps the body of the function. The body of the function is wrapped in a function object that can be called later to compute the next element of the sequence.

When you later call a sequence function like first, rest, map , or filter on the lazy sequence object, Clojure will call the function object generated by lazy-seq to compute the next element of the sequence. This will cause the body of the lazy-seq function to be executed, if it has not already been executed, and one or more elements of the sequence to be realized.

Once an element of the sequence has been realized, it is cached, so that subsequent calls to sequence functions like first, rest, map, or filter can return the same value without recomputing it. This caching feature is important and can be useful for avoiding unnecessary computation when dealing with large or potentially infinite sequences.

One important thing to note is that lazy sequences are realized incrementally, one element at a time. This means that if you call a sequence function that requires the entire sequence to be realized, like count or reduce, then Clojure will need to realize the entire sequence before it can return a value. This can cause performance issues or out-of-memory errors if you try to realize an infinite or very large sequence in this way. Clojure also has some optimisations that will compute 32 items of lazy-seq at once.

Alternatives to lazy-seq in Clojure

You don't need to use lazy-seq directly to work with lazy sequences in Clojure. There are alternatives to lazy-seq in Clojure, each with its own advantages and limitations. Here are some of the most commonly used alternatives:

iterate: iterate is a built-in function that returns a lazy sequence of values generated by applying a function to an initial value repeatedly. For example, (iterate inc 0) generates an infinite sequence of natural numbers.
repeatedly: repeatedly is a built-in function that returns a lazy sequence of values generated by calling a function repeatedly. For example, (repeatedly 10 #(rand-int 10)) generates a sequence of 10 random integers between 0 and 9.
range: range is a built-in function that returns a lazy sequence of numbers in a specified range. For example, (range 1 10) generates a sequence of integers from 1 to 9.
mapcat: mapcat is a higher-order function that applies a function to each element of a sequence and concatenates the results into a single sequence. For example, (mapcat #(range 1 %) [2 3 4]) generates a sequence of integers from 1 to 2, then from 1 to 3, and finally from 1 to 4.
concat: concat is a function that concatenates two or more sequences into a single sequence. For example, (concat [1 2 3] [4 5 6]) generates a sequence of integers from 1 to 6.

Each of these alternatives has its own use cases and trade-offs. For example, iterate and repeatedly are good for generating sequences of values based on a function, while range is good for generating sequences of consecutive numbers. mapcat and concat are good for combining multiple sequences into a single sequence. However, none of these alternatives can match the flexibility and power of lazy-seq for generating complex and potentially infinite sequences of values.

Also read about Stuart Sierra's advice Clojure Dont's: Concat

Summary

Lazy evaluation is a technique used in functional programming languages to delay the evaluation of expressions until their values are actually needed. In Clojure, lazy evaluation can be implemented using lazy sequences.

You can use the lazy-seq macro to create a lazy sequence in Clojure. It takes a body of code that generates the next element of the sequence and returns a new lazy sequence that evaluates the body of code only when necessary.

Lazy evaluation is useful for

Efficient processing of potentially infinite sequences of values
Dealing with large data sets
Performing complex computations on streams of data.

Two important features of lazy sequences in Clojure are

Delayed computation until in is actually needed
Caching of computed values

Important note about lazy-seq, it does 32 at a time not just one as an optimisation.

References

Allow or Deny list with AWS resources-based policies

Mon, 17 Jan 2022 22:42:16 GMT

This post discusses how to create a Resource based IAM policy that Allow or Deny actions for a list of Principals. Here I will show some examples that are recommended and also some examples that are not recommended. We also talk about AWS IAM Policy Simulator a little bit.

As you know AWS IAM (Identity and Access Management) policies define and manage permissions for resources in AWS. They allow you to specify which actions a user, group, or role can perform on a particular resource or set of resources. IAM Policies are usually attached to a Principal like user, group or role for managing the permission.

Resource based IAM Policies

Resource-based policies are a type of IAM policy that are attached to AWS resources, such as an S3 bucket, instead of an IAM identity like a user or role. These policies define the permissions that other AWS identities have on the resource they are attached to.

Resource-based policies are particularly useful when you want to grant cross-account access to AWS resources. By attaching a resource-based policy to a resource, you can specify which AWS accounts, IAM roles/users/groups are allowed to access that resource, and what actions they are allowed to perform.

AWS resources that support resource-based policies include S3 buckets, Lambda functions, API Gateway APIs, and SQS queues, among others. You can find the full list of resources that support resource-based polices here.

Resource-based policies can be used to grant fine-grained access control to your AWS resources, while also maintaining security and compliance.

Allow list and Deny list

By allow list we mean a list of Principals that can exclusively allowed to perform actions on certain resources. And by Deny list we mean a list of Principals that are exclusively denied permission on certain resources.

Let us see how this can be done.

To add an allow or deny list of principals to a resource-based policy, you can use the Principal element in the policy document.

Here's an example of how to allow a specific AWS account ID to access an S3 bucket using a resource-based policy:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "AllowAccountAccess",
    "Effect": "Allow",
    "Principal": {
      "AWS": "arn:aws:iam::123456789012:root"
    },
    "Action": "s3:*",
    "Resource": "arn:aws:s3:::my-bucket/*"
  }]
}

In this example, the Principal element specifies the AWS account ID that is allowed to access the S3 bucket. The Effect element is set to Allow to indicate that access is permitted, and the Action element specifies the actions that are allowed on the S3 bucket. This policy must be attached to the my-bucket and it allows for cross account access from the account 123456789012.

To add a deny list of principals, you can add a second statement to the policy with the Effect element set to Deny.

Here's an example of how to deny access to a specific IAM user:

{
  "Version": "2012-10-17",
  "Statement": [{
      "Sid": "AllowAccountAccess",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789012:root"
      },
      "Action": "s3:*",
      "Resource": "arn:aws:s3:::my-bucket/*"
    },
    {
      "Sid": "DenyUserAccess",
      "Effect": "Deny",
      "Principal": {
        "AWS": "arn:aws:iam::123456789012:user/example-user"
      },
      "Action": "s3:*",
      "Resource": "arn:aws:s3:::my-bucket/*"
    }
  ]
}

The above policy allows all Principals in account 123456789012 except example-user to perform actions on my-bucket.

Note that an explicit Deny in a policy overrides any Allow. This Deny also takes precedence even if the Principal's policy explicitly allows actions on the S3 bucket. Read about IAM policy evaluation here.

Double negation (not recommended)

Using double negation in a resource-based policy allows you to create an allow list of AWS principals that are explicitly allowed to access a resource, while denying access to all other principals.

But it is confusing and harder to maintain.

Here's an example of how to use double negation to create an allow list of AWS account IDs:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "AllowAccountAccess",
    "Effect": "Deny",
    "NotPrincipal": {
      "AWS": [
        "arn:aws:iam::123456789012:root",
        "arn:aws:iam::987654321098:root"
      ]
    },
    "Action": "s3:*",
    "Resource": "arn:aws:s3:::my-bucket/*"
  }]
}

In this example, the NotPrincipal element is used to specify a list of AWS account IDs that are explicitly denied access to the S3 bucket. Here the Deny Effect in the statement, denies access is to all other AWS principals, effectively creating an allow list of the specified AWS account IDs.

As you see double negation policies can be more complex to read and maintain, so it's important to test and validate the policy to ensure it's working as intended.

It is very important to keep the policy up to date as AWS principals and resource access requirements change over time. Next example shows better way.

Using Condition

Here's an example of how to use a condition to allow a specific AWS account ID to access an S3 bucket, while denying access to all other Principals.

{
  "Version": "2012-10-17",
  "Statement": [{
      "Sid": "AllowAccountAccess",
      "Effect": "Allow",
      "Principal": "*",
      "Action": "s3:*",
      "Resource": "arn:aws:s3:::my-bucket/*",
      "Condition": {
        "StringEquals": {
          "aws:PrincipalAccount": "123456789012"
        }
      }
    },
    {
      "Sid": "DenyAllOtherAccess",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:*",
      "Resource": "arn:aws:s3:::my-bucket/*"
    }
  ]
}

In this example, the Condition element is used to specify that the policy should only apply to the AWS account with the ID of 123456789012.

Note that this approach may require updating the policy if additional AWS principals need access to the resource in the future, as the StringEquals element does not allow lists. However, it can be a simpler way to manage an allow list that only includes one or a few AWS principals, rather than using double negation in a resource-based policy.

List of users or roles

The AWS element in the Principal block can accept a list instead a single item. We can use this to create a resource-based policy that allows access only for a specific list of users or roles as principals.

Here's an example of a resource-based policy that allows access only to a specific list of IAM users and roles:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "AllowSpecificUsersAndRoles",
    "Effect": "Allow",
    "Principal": {
      "AWS": [
        "arn:aws:iam::123456789012:user/user1",
        "arn:aws:iam::123456789012:role/role1",
        "arn:aws:iam::123456789012:user/user2"
      ]
    },
    "Action": "s3:*",
    "Resource": "arn:aws:s3:::my-bucket/*"
  }]
}

If you want to deny access to all other AWS principals, you can add a second statement with Effect set to Deny, as follows:

{
  "Version": "2012-10-17",
  "Statement": [{
      "Sid": "AllowSpecificUsersAndRoles",
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::123456789012:user/user1",
          "arn:aws:iam::123456789012:role/role1",
          "arn:aws:iam::123456789012:user/user2"
        ]
      },
      "Action": "s3:*",
      "Resource": "arn:aws:s3:::my-bucket/*"
    },
    {
      "Sid": "DenyAllOtherAccess",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:*",
      "Resource": "arn:aws:s3:::my-bucket/*"
    }
  ]
}

In this case, the second statement denies access to all other AWS principals, thus creating an exclusive allow list of Principals that have access to this S3 bucket.

The above policy is a great way for having explicit security policy for compliance purposes.

Double negation for exclusivity (not recommended)

To make the double negation policy equivalent to the previous resource-based policy, you can modify it to explicitly deny access for the specified list of IAM users and roles, as follows:

{
  "Version": "2012-10-17",
  "Statement": [{
      "Sid": "AllowSpecificUsersAndRoles",
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::123456789012:user/user1",
          "arn:aws:iam::123456789012:role/role1",
          "arn:aws:iam::123456789012:user/user2"
        ]
      },
      "Action": "s3:*",
      "Resource": "arn:aws:s3:::my-bucket/*"
    },
    {
      "Sid": "DenyAllOtherAccess",
      "Effect": "Deny",
      "NotPrincipal": {
        "AWS": [
          "arn:aws:iam::123456789012:user/user1",
          "arn:aws:iam::123456789012:role/role1",
          "arn:aws:iam::123456789012:user/user2"
        ]
      },
      "Action": "s3:*",
      "Resource": "arn:aws:s3:::my-bucket/*"
    }
  ]
}

In this modified policy, the first statement explicitly allows access for the specified list of IAM users and roles, and the second statement denies access for all other AWS principals.

This is completely unnecessary, lengthy and confusing, so it is not recommended. The previous example with wildcard(*) deny for all does the same and is much cleaner.

One more example that is not recommended

What do you think the following policy will do?

{
  "Version": "2012-10-17",
  "Statement": [{
      "Sid": "DoNotDenyPrincipals",
      "Effect": "Deny",
      "NotPrincipal": {
        "AWS": [
          "arn:aws:iam::123456789012:user/user1",
          "arn:aws:iam::123456789012:role/role1",
          "arn:aws:iam::123456789012:user/user2"
        ]
      },
      "Action": "s3:*",
      "Resource": "arn:aws:s3:::my-bucket/*"
    },
    {
      "Sid": "AllowAllOtherAccess",
      "Effect": "Allow",
      "Principal": "*",
      "Action": "s3:*",
      "Resource": "arn:aws:s3:::my-bucket/*"
    }
  ]
}

In this policy, the first statement denies access to all AWS principals that are not in the list of IAM users and roles specified in the policy. However, it does not explicitly allow access for the specific list of IAM users and roles, as the previous resource-based policy does.

The second statement allows access for all other AWS principals, including those that are not in the specified list of IAM users and roles. This is not what we want because it allows access for all other AWS principals, not just those explicitly specified in the policy.

To make this policy equivalent to the previous resource-based policy that explicitly allows access to a specific list of IAM users and roles, we can modify it as follows:

{
  "Version": "2012-10-17",
  "Statement": [{
      "Sid": "AllowSpecificUsersAndRoles",
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::123456789012:user/user1",
          "arn:aws:iam::123456789012:role/role1",
          "arn:aws:iam::123456789012:user/user2"
        ]
      },
      "Action": "s3:*",
      "Resource": "arn:aws:s3:::my-bucket/*"
    },
    {
      "Sid": "DenyAllOtherAccess",
      "Effect": "Deny",
      "NotPrincipal": {
        "AWS": [
          "arn:aws:iam::123456789012:user/user1",
          "arn:aws:iam::123456789012:role/role1",
          "arn:aws:iam::123456789012:user/user2"
        ]
      },
      "Action": "s3:*",
      "Resource": "arn:aws:s3:::my-bucket/*"
    }
  ]
}

In this modified policy, the first statement explicitly allows access for the specified list of IAM users and roles, and the second statement denies access for all other AWS principals.

Although this policy will work as expected, I wouldn't recommend using this in production due to the double negation in it.

How can we confirm if this works?

To confirm that the policy works as intended, you can test it with different IAM users and roles that are either in or not in the list of specified IAM users and roles.

You will need to sign in as the user or assume the role that is listed in the policy and perform actions on my-bucket to test it manually. The action should be allowed this time.

You should also test the opposite with a user or role that is not listed in the policy for making sure exclusivity is in place as expected.

You can also use the AWS Policy Simulator to test the policy with different scenarios and confirm that it works as intended. The simulator can help you identify any issues or errors in your policy.

To use the Policy Simulator, you can select a policy you want to test, specify a set of actions, resources, and conditions, and then run the simulation. The simulator will then show you the results of the simulation, indicating whether the policy allows or denies the specified actions.

You can also use the simulator to test policies with different roles or users, and to see how policy evaluation works in different scenarios. This can be a useful way to identify any unintended consequences of your policy, or to identify areas where you need to refine or improve your policy.

Overall, the Policy Simulator is a powerful tool that can help you create and test effective IAM policies, and ensure that your AWS resources are properly secured.

Here is nice blog that shows how to test resource-based policy using the policy simulator: Testing an S3 policy using the AWS IAM Simulator

Which style is better for writing and maintaining this kind of policy?

The style vastly depends on your preferences and the requirements of your specific use case.

That being said, it is recommended to use the most straightforward and clear approach that achieves the desired result. In this case, the approach that explicitly lists the allowed principals and uses a deny statement to block all other access is the most clear and simple to understand.

Using double negation is complex and harder to read and understand for someone who is not familiar with the policy.

The most important thing is to choose an approach that is clear, concise, and meets the security requirements of your use case. You should also consider the long-term maintainability of the policy, and choose an approach that will be easy to update and modify as your requirements change over time.

Summary

Explicitly list the allowed principals and use a wildard(*) deny statement to block all others:

{
  "Version": "2012-10-17",
  "Statement": [{
      "Sid": "AllowSpecificUsersAndRoles",
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::123456789012:user/user1",
          "arn:aws:iam::123456789012:role/role1",
          "arn:aws:iam::123456789012:user/user2"
        ]
      },
      "Action": "s3:*",
      "Resource": "arn:aws:s3:::my-bucket/*"
    },
    {
      "Sid": "DenyAllOtherAccess",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:*",
      "Resource": "arn:aws:s3:::my-bucket/*"
    }
  ]
}

Explicitly list the denied principals and use a wildcard (*) allow statement to allowed all others:

{
  "Version": "2012-10-17",
  "Statement": [{
      "Sid": "AllowAllAccess",
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": "s3:*",
      "Resource": "arn:aws:s3:::my-bucket/*"
    },
    {
      "Sid": "AllowAccountAccess",
      "Effect": "Deny",
      "Principal": {
        "AWS": [
          "arn:aws:iam::123456789012:user/user3"
        ]
      },
      "Action": "s3:*",
      "Resource": "arn:aws:s3:::my-bucket/*"
    }
  ]
}

Hope this information is helpful!

References

Migrated homepage from Gridsome to NuxtJS

Fri, 15 Oct 2021 22:29:32 GMT

Moving my homepage to Gridsome and Vue was great choice because of the simplicity of VueJS. I'm happy with the choice of Vue.

However, Gridsome has been lagging lagging behind on the new advances available in Vue3. Also I realise I'm not using any advanced features from Gridsome for my homepage.

I've been looking at NuxtJS for sometime now and it looks really good choice. NuxtJS seems a lot simpler than Gridsome. I also wanted to start using TypeScript.

The migration was simple enough as seen in this pr. I also a wrote a new Component in TypeScript that mimics a typewriter behaviour.

New homepage with Vue and Gridsome

Sun, 29 Nov 2020 05:13:31 GMT

It's been so long since I've re-written my homepage.

It started orginially as a plain html file, custom css, bootstrap, fontawesome, and google font.

This is how the <head></head> looked like.

<link rel="stylesheet" type="text/css" href="https://cdn.jsdelivr.net/fontawesome/4.3.0/css/font-awesome.min.css" />
<link rel="stylesheet" type="text/css" href="https://cdn.jsdelivr.net/bootstrap/3.3.2/css/bootstrap.min.css" />

<link rel="stylesheet" type="text/css" href="https://fonts.googleapis.com/css?family=Lora:400,400italic,700,700italic|Bitter:400&subset=latin,latin-ext" media="all" />
<link rel="stylesheet" type="text/css" href="css/style.css" />

As you can imagine, that's a a lot of static assets to download for a simple homepage. Although hopefully fontawesome, bootstrap, and google fonts are already cached is most browsers!.

Here I had to write the content in pure html and was hard to maintain.

Things like privacy-policy and terms of use were very difficult to maintain as html files.

New and shiny

The new homepage is built using gridsome which is built on top of vue.js. CSS in the new homepae is handled by tailwindcss.

Now the content is written in markdown and gridsome does the heavy lifting of creating static html and optimized javascript. Tailwind and Purgecss creates optimizd css files.

Python stdin stderr logging

Fri, 29 May 2020 10:22:18 GMT

Logging in Python is quite straight forward, except for by the BasicConfig writes to stderr and not to stdout

import logging

log_format = '%(name)s - %(levelname)s - %(message)s'
logging.basicConfig(format=log_format)
log = logging.getLogger(__name__)
log.warning('I print to stderr by default')
log.info('I print to stderr by default')
log.error('I print to stderr by default')

This can cause info logs to be written to stderr and in AWS Glue it ends up in the error logs.

Setup logs to stdout

import logging
import sys

root = logging.getLogger()
root.setLevel(logging.DEBUG)

handler = logging.StreamHandler(sys.stdout)
handler.setLevel(logging.DEBUG)
log_format = '%(name)s - %(levelname)s - %(message)s'
formatter = logging.Formatter(log_format)
handler.setFormatter(formatter)
root.addHandler(handler)

log =logging.getLogger(__name__)
log.info("Writes to stderr by default')

Using print()

Clearly using logging is a but complicated for some usecases.

print() in python writes to stdout by default. But we can make it write to stderr.

#!/usr/bin/env python
import sys
print('this prints to stdout')
print("this prints to stderr", file=sys.stderr)

AWS Glue Python Shell logs

You can use print statements in your Glue Python Shell job for logging. Glue captures stdout and stderr by default.

There is no need to setup a logger in Glue Python Shell job.

Redirecting stderr and stdout

Most shells will show you stdout and stderr logs in the console. You can redirect error logs and stdout logs into different files.

Use 1> to redirect stdout and 2> to redirect stderr.

>> can be used to append operation

myscript 2> /tmp/error.log 1> /tmp/out.log # overwrite
myscript 2>> /tmp/error.log 1>> /tmp/out.log # append

References

Query AWS Athena with Emacs using jdbc

Thu, 28 Mar 2019 01:58:23 GMT

If you haven't checked out ejc-sql, you should now.

Official JDBC connection setup using jdbc drivers is available here - https://github.com/kostafey/ejc-sql#create-connections-manualy

It has support for

MySQL
MariaDB
MS SQL Server
Oracle
H2
SQLite
PostgreSQL
Informix
Presto

And also allows you to add custom jdbc drivers using ejc-jdbc-drivers

You can customize artifacts and their versions used as JDBC drivers for each database type in Leiningen format in ejc-jdbc-drivers custom variable.

Here is how I setup it up

;; setup with use-package
(use-package ejc-sql
  :commands
  (ejc-create-connection ejc-connect ejc-set-column-width-limit ejc-set-max-rows)
  :init
  (setq ejc-set-rows-limit 1000
        ejc-result-table-impl 'orgtbl-mode ;; 'ejc-result-mode
        ejc-use-flx t
        ejc-flx-threshold 3
        nrepl-sync-request-timeout 30)
  ;; enable auto complete
  (add-hook 'ejc-sql-minor-mode-hook
            (lambda ()
              (auto-complete-mode t)
              (ejc-ac-setup))))

;; custome interactive function to create connection
(defun my/setup-ejc-connections ()
  (interactive)
  (ejc-set-column-width-limit 72)
  ;; add athena jdbc driver to the `ejc-jdbc-drivers`
  (plist-put ejc-jdbc-drivers
            "awsathena"
            [com.simba.athena/athena-jdbc "2.0.9"])
  (ejc-create-connection "athena-prod"
                        :dbtype "awsathena"
                        :classpath "~/.m2/repository/com/simba/athena/athena-jdbc/2.0.9/athena-jdbc-2.0.9.jar"
                        :classname "com.simba.athena.jdbc.Driver"
                        :subprotocol "awsathena"
                        :subname "//athena.eu-west-1.amazonaws.com:443/my_database;S3OutputLocation=s3://my-athena-bucket/"
                        :user (getenv "AWS_ACCESS_KEY_ID")
                        :password (getenv "AWS_SECRET_ACCESS_KEY")))
  ;; You can also use connectin-uri like below
  ;; :connection-uri (concat "jdbc:awsathena://athena.eu-west-1.amazonaws.com:443/my_database;"
  ;;                         "User=" (getenv "AWS_ACCESS_KEY_ID") ";"
  ;;                         "Password=" (getenv "AWS_SECRET_ACCESS_KEY") ";"
  ;;                         "S3OutputLocation=" "s3://my-athena-bucket/"))

I mostly use this in org-mode buffer so that I can analyse data and make notes about it.

EDIT 08/04/2021

You may also simplify connection configuration using Profile attribute in :connection-uri with the latest Athena JDBC driver.

This also helps if you are using AWS STS temporary tokens for authenticating with AWS.

(ejc-create-connection
 "athena-prod"
 :dbtype "awsathena"
 :classpath "~/.m2/repository/com/simba/athena/athena-jdbc/2.0.16.1000/athena-jdbc-2.0.16.1000.jar"
 :classname "com.simba.athena.jdbc.Driver"
 :connection-uri "jdbc:awsathena://AwsRegion=eu-west-1;Profile=MyProfile;S3OutputLocation=s3://aws-athena-query-results-xxxxxx-eu-west-1/")

Querying AWS Athena with Clojure using HugSQL

Fri, 08 Mar 2019 01:44:13 GMT

HugSQL is handsdown the best experience I've had with relational database frameworks.

I've used it with Postgres and AWS Athena. Getting started with HugSQL and postgres is simple as it is documented in https://hugsql.org.

This blog is about HugSQL configuration to query Athena. Since you are here, I assume you already know HugSQL or you are using it.

Athena JDBC my Simba

AWS provides a JDBC driver for Athena. Note that this driver is working on top of http API's provided by AWS, so we don't have jdbc connection session that is open all the time. This has some limitations.

You can read about Athena JDBC driver features in this link https://www.simba.com/drivers/athena-odbc-jdbc/

You can find the jdbc driver documentation here.

Below are the two types of configurations that worked for me with hugsql version [com.layerware/hugsql "0.5.1"]

Install the drivers

Install Athena JDBC Driver in your local repository

Athena JDBC driver is not directly available from Maven central, although third party one's exist.

The official jar can be downloaded and installed in your local maven repository with following.

# Download official athena jdbc driver
wget https://s3.amazonaws.com/athena-downloads/drivers/JDBC/SimbaAthenaJDBC_2.0.9/AthenaJDBC42_2.0.9.jar

# Install in local maven repository
mvn install:install-file \
-Dfile=AthenaJDBC42_2.0.9.jar \
-DgroupId=com.simba.athena \
-DartifactId=athena-jdbc \
-Dversion=2.0.9 \
-Dpackaging=jar

Configuration

Using dbhost

(def athena-db {:dbtype "awsathena"
                :host "athena.amazonaws.com"
                :port 443
                :dbname "my_database"
                :classname "com.simba.athena.jdbc.Driver"
                :AwsCredentialsProviderClass "com.amazonaws.auth.EnvironmentVariableCredentialsProvider"
                :AWSRegion "eu-west-1"
                :S3OutputLocation "s3://athena-query-results-eu-west-1/"
                :LogLevel 3})

Using subprotocol

(def athena-db {:dbtype "awsathena"
                :subprotocol "awsathena"
                :subname "//athena.eu-west-1.amazonaws.com:443/my_database;"
                :classname "com.simba.athena.jdbc.Driver"
                :AwsCredentialsProviderClass "com.amazonaws.auth.EnvironmentVariableCredentialsProvider"
                :S3OutputLocation "s3://athena-query-results-eu-west-1/"
                :LogLevel 4})

This driver will look for credentials in the AWS credentials prodiver chain.

In my case I have the credentials in environment variables, and I rotate them periodically.

Running a query

-- :name get-10-items-from-my-table
-- :doc Get 10 items for my_table
SELECT * FROM
my_database.my_table
LIMIT 10

(ns query-runner
  (:require [clojure.string :as str]
            [hugsql.core :as hugsql]
            [clj-time.jdbc]))

(def map-of-db-fns (hugsql/map-of-db-fns "myqueries.sql"))

(when-some [query-fn (get-in map-of-db-fns [:get-10-items-from-my-table :fn])]
  (query-fn athena-db))

That should do!

Let me know if this is useful in comments, thanks.

List Destructuring

Wed, 04 Apr 2018 00:05:01 GMT

I was reading the Clojure for Brave and True book, when I stumbled upon destructring feature in Clojure.

Clojure does it neatly! https://clojure.org/guides/destructuring

This example is taken directly from the book.

(defn chooser
  [[first-choice second-choice & unimportant-choices]]
  (println (str "Your first choice is: " first-choice))
  (println (str "Your second choice is: " second-choice))
  (println (str "We're ignoring the rest of your choices. "
                "Here they are in case you need to cry over them: "
                (clojure.string/join ", " unimportant-choices))))

(chooser ["Marmalade", "Handsome Jack", "Pigpen", "Aquaman"])
; => Your first choice is: Marmalade
; => Your second choice is: Handsome Jack
; => We're ignoring the rest of your choices. Here they are in case \
;     you need to cry over them: Pigpen, Aquaman

Now, I wanted to see how other languages support this.

Scala

Scala has powerful pattern matching abilities for destructuring a list.

val first::second::rest = List(1,2,3,4,5,6,7,8,9,10)
// first: Int = 1
// second: Int = 2
// rest: List[Int] = List(3, 4, 5, 6, 7, 8, 9, 10)

Scala's pattern matching is so powerful that you can go further with it. For example you can find the second last element, or third last element and what ever is inbetween them.

val first :: second:: (inbetween :+ thirdLast :+ secondLast :+ last) = List(1,2,3,4,5,6,7,8,9,10)
// first: Int = 1
// second: Int = 2
// inbetween: List[Int] = List(3, 4, 5, 6, 7, 8)
// secondLast: Int = 9
// last: Int = 10

Ruby

As you know Ruby has this ability as well!

> first, second = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
=> [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
> first
=> 1
> second
=> 2

> first, second, *rest = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
=> [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
> first
=> 1
> second
=> 2
> rest
=> [3, 4, 5, 6, 7, 8, 9, 10]

> first, second, *inbetween, secondLast, last = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
=> [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
> last
=> 10
> secondLast
=> 9
> inbetween
=> [3, 4, 5, 6, 7, 8]
> first
=> 1
> second
=> 2

Python

Python also does it beautifully.

[first, second, rest] = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
[first, second, *inbetween, secondLast, last] = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
# >>> secondLast
# 9
# >>> last
# 10
# >>> second
# 2
# >>> first
# 1
# >>> inbetween
# [3, 4, 5, 6, 7, 8]

Javascript

Javascript also support's destructuring.

const [first, second, ...rest] = [1, 2, 3, 4, 5, 6, 7];

Java

Are you kidding me? 😄

References

How to check OS and version in Linux

Mon, 04 Dec 2017 14:19:55 GMT

From time to time you end up in a machine running somewhere in the cloud and wonder what OS is running there. You know it's a flavour of Linux, but you don't know which one. Worry not my friend, this post will help you.

Kernel Version

If you want kernel version information, use uname(1).

$ uname -a
Linux localhost 4.4.0-101-generic #124-Ubuntu SMP Fri Nov 10 18:29:59 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

$ uname -a
Linux 10.0.2.15 4.13.15-300.fc27.x86_64 #1 SMP Tue Nov 21 21:10:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Distribution Information

Information about the distribution varies depending on the distribution. If your distribution supports Linux Standard Base, then we can find that information using lsb_release. It can also be found from *-release files in /etc/ directory.

$ lsb_release -a
No LSB modules are available.
Distributor ID:  Ubuntu
Description:  Ubuntu 16.04.3 LTS
Release:  16.04
Codename:  xenial

On Fedora you may need to install lsb_release.

$ sudo yum install lsb_release -a
$ lsb_release -a
LSB Version:  :core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch
Distributor ID:  Fedora
Description:  Fedora release 27 (Twenty Seven)
Release:  27
Codename:  TwentySeven

Release files

Ubuntu has two release files /etc/lsb-release and /etc/os-release

$ cat /etc/*-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.3 LTS"
NAME="Ubuntu"
VERSION="16.04.3 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.3 LTS"
VERSION_ID="16.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial

Fedoro has /etc/fedora-release, /etc/os-release, /etc/redhat-release, /etc/system-release

$ cat /etc/*-release
 sudo cat /etc/*release
Fedora release 27 (Twenty Seven)
NAME=Fedora
VERSION="27 (Cloud Edition)"
ID=fedora
VERSION_ID=27
PRETTY_NAME="Fedora 27 (Cloud Edition)"
ANSI_COLOR="0;34"
CPE_NAME="cpe:/o:fedoraproject:fedora:27"
HOME_URL="https://fedoraproject.org/"
SUPPORT_URL="https://fedoraproject.org/wiki/Communicating_and_getting_help"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=27
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=27
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
VARIANT="Cloud Edition"
VARIANT_ID=cloud
Fedora release 27 (Twenty Seven)
Fedora release 27 (Twenty Seven)

Some extra places to look into for Ubuntu

$ cat /etc/issue.net
Ubuntu 16.04.3 LTS

$ cat /etc/debian_version
stretch/sid

References:

Install additional Python packages on Azure HDInsights cluster

Tue, 14 Nov 2017 12:51:35 GMT

An HDInsights cluster consists of several nodes. A PySpark script may run on any one of those nodes.

In order to install a package on HDInsights we need to use Script Actions

Use Script Action

Script Action allows you to run a shell script on all nodes of an Azure cluster. You may run the script action when you create the cluster or while the cluster is running.

You may also persist a Script Action in the cluster, so that it is run on every new node that is added to the cluster.

Pip install script action

#!/usr/bin/env bash
/usr/bin/anaconda/bin/pip install --upgrade pip $@

This script will install and update the given packages in the parameters section.

For example, you may install a new version of Plotly to use in your Jupyterhub notebooks.

Upload your Script Action

Your script action must be in a location accessible to the cluster. It could be anywhere, but let's upload it to storage account.

Upload your Script Action to a Storage Account that your chosen HDInsights have access to.

Bash script URI https://<my-storage-account>.blob.core.windows.net/script-actions/pip-install-upgrade-packages.sh

Steps

Select the cluster that you want to run the Script action in Azure portal
On the blade under Configuration, you can find Script Actions
Select that and then click Submit New
Select custom
Give a name
Select nodes to install
In the parameters section give list of packages to install separated by spaces

ARM Template

We can also add this script to HDInsights ARM template in the `computeProfile` as `scriptActions`.

"computeProfile": {
  "roles": [
    {
      "name": "headnode",
      "scriptActions": [
        {
          "name": "pip-install-upgrade-packages",
          "uri": "https://<my-storage-account>.blob.core.windows.net/script-actions/pip-install-upgrade-packages.sh",
          "parameters": "plotly"
        }
      ]
    },
    {
      "name": "workernode",
      "scriptActions": [
        {
        "name": "pip-install-upgrade-packages",
          "uri": "https://<my-storage-account>.blob.core.windows.net/script-actions/pip-install-upgrade-packages.sh",
          "parameters": "plotly"
        }
      ]
    }
  ]
}

NB: You may create other Script Actions to do any kind of customisations on your HDInsights Cluster

References

Hortonworks Sandbox on Ubuntu using docker

Thu, 02 Nov 2017 16:11:53 GMT

In a previous post, I talked about installing Hortonworks Sandbox on a Mac machine. That was straight forward.

In this blog ’m going to talk about installing the Hortonworks Sandbox manually on a Linux VM.

A little background about Docker base device size

The first time docker is started, it sets up a base device with a default size of 10GB. All future images and containers would be a snapshot of this base device.

Base size is the maximum size that a container/image can grow to. By default docker limit containers to 10G. In Devicemapper, new container/images take zero size and grow up to the maximum size. Changing the base size will not actually change the physical usage of containers unless they grow larger than 10 GB.

You can view this information with the following command

sudo docker info

Since Hortonworks image is greater than 10GB, we need to increase the storage base size. We can do this by changing the storage-driver to overlay and increasing the dm.bazesize.

Increase the docker base device size on Ubuntu

For this we need to set storag-opt=dm.basesize and change storage-diver to overlay.

Assuming we have Systemd based setup

First create directory for customising docker service

sudo mkdir /etc/systemd/system/docker.service.d/

Edit /etc/systemd/system/docker.service.d/docker.conf

sudo vi /etc/systemd/system/docker.service.d/docker.conf

[Service]
ExecStart=
ExecStart=/usr/bin/dockerd --graph="/mnt/docker-data" --storage-driver=overlay --storage-opt=dm.basesize=30G

Here the first line with empty ExecStart is necessary to clear previous configurations. Also note that we are increasing the dm.basesize to 30G

IMPORTANT: Make sure there is enough space in /mnt/docker-data

Reload systemd daemon and restart docker

After editing docker service definition for systemd we need to releoad systemd daemon and restart docker, for the new settings to take effect.

sudo systemctl daemon-reload
sudo systemctl restart docker

If you check sudo docker info, Storage Driver should be overlay and Docker Root Dir should be /mnt/docker-data

Now we have setup docker to work with Hortonworks Sandbox.

Download the Hortonworks Sandbox Docker image

Download the docker image. It’s a huge one 12GB, use high bandwidth network :)

wget https://downloads-hortonworks.akamaized.net/sandbox-hdp-2.6.1/HDP_2_6_1_docker_image_28_07_2017_14_42_40.tar

Download the Hortonworks Script

wget https://raw.githubusercontent.com/hortonworks/data-tutorials/master/tutorials/hdp/sandbox-port-forwarding-guide/assets/start-sandbox-hdp.sh

Load the sandbox image

docker load -i HDP_2_6_1_docker_image_28_07_2017_14_42_40.tar

This is a heavy task. Go get your coffee now because it will take some time to load the docker image.

This step would fail if you did not increase the base device size.

After it is loaded you should see sandbox-hdp in your docker images list.

Update /etc/hosts file

Add the following to /etc/hosts

127.0.0.1   localhost   sandbox.hortonworks.com

Run the Hortonworks Sandbox startup script

chmod +x ./start-sandbox-hdp.sh
./start-sandbox-hdp.sh

Again this will take some time. Once the script completes you should be able to access the sandbox.

Access Hortonworks Sandbox

Go to https://sandbox.hortonworks.com:8888/

This Sandbox comes with a lot of components like Ambari, Ranger, Hive, Spark etc. installed and configured.

References

SSH config tips

Mon, 30 Oct 2017 16:21:55 GMT

SSH config file

~/.ssh/config file can be very useful to configure settings for your ssh connections.

For example you may configure default User and IdentityFile for a domain using.

Host example.com
   IdentityFile /path/to/identityfile
   User defaultUserName

Multiple Identities on same server

I use trick for configuring multiple identities on Bitbucket server.

Host personal.bitbucket.org
   HostName bitbucket.org
   User git
   IdentityFile ~/.ssh/id_rsa

Host work.bitbucket.org
   HostName bitbucket.org
   User git
   IdentityFile ~/.ssh/id_rsa_work

And you need to use personal.bitbucket.org and work.bitbucket.org when you want to ssh into them.

Bitbucket has wild card domain redirection, that’s why this works for Bitbucket.

For other domains, you may need to add an entry into /etc/hosts.

Specify Port

If you have configured your SSH server to use a different port instead of 22, lets say 2222

Host my.server.fqdn
   Port 2222

Force password authentication

You can force Password Authentication by using

ssh -o PreferredAuthentications=password -o PubkeyAuthentication=no user@host

Port forwarding

This is my favourite of all.

ssh -L 9000:mydb.host:9000 user@example.com

This can be very useful for accessing a port behind a firewall, may be for database.

If you are using AWS EMR or Hortworks Sanbdox, then you will need access to more than one port through ssh.

Using -L command line option can be very lengthy, instead you could use LocalForward directive in the ~/.ssh/config

# This will allow us to use port 8080 on the local machine
# in order to access example.com at port 80 from the remote machine
Host local_to_remote
    LocalForward 8080 example.com:80

# This will allow us to offer access to internal.com at port 443
# to the remote machine through port 7777 on the other side
Host remote_to_local
    RemoteForward 7777 internal.com:443

References

Setting up Hortonworks Sandbox on Mac using Docker

Thu, 26 Oct 2017 15:49:57 GMT

Hortonworks provides an official Data Platform Sandbox docker image that can help you learn the Hortonworks stack. I recently setup a Hortonworks Sandbox on my macOS.

Increase Docker memory for macOS

Hortonworks Sandbox is a heavy container, with lots of components running on it. They recommend at least 8GB of RAM.

Open Docker preferences and select the Advanced tab. Increase the dedicated memory to at least 8GB of RAM or more.

Configure Docker Memory

Download the Hortonworks Sandbox Docker image

Download the docker image. It’s a huge one 12GB, use high bandwidth network :)

wget https://downloads-hortonworks.akamaized.net/sandbox-hdp-2.6.1/HDP_2_6_1_docker_image_28_07_2017_14_42_40.tar

Download the Hortonworks Script

wget https://raw.githubusercontent.com/hortonworks/data-tutorials/master/tutorials/hdp/sandbox-port-forwarding-guide/assets/start-sandbox-hdp.sh

Load the sandbox image

docker load -i HDP_2_6_1_docker_image_28_07_2017_14_42_40.tar

This is a heavy task. Go get your coffee now because it will take some time to load the docker image.

After it is loaded you should see sandbox-hdp in your docker images list.

Update /etc/hosts file

Add the following to /etc/hosts

127.0.0.1   localhost   sandbox.hortonworks.com

Run the Hortonworks Sandbox startup script

chmod +x ./start-sandbox-hdp.sh
./start-sandbox-hdp.sh

Again this will take some time… ⏲

Access Hortonworks Sandbox

Go to https://sandbox.hortonworks.com:8888/

This Sandbox comes with a lot of components like Ambari, Ranger, Hive, Spark etc. installed and configured.

References

Hortonworks Sandbox and install guide

Function to change the extension of current file in Emacs lisp

Sun, 25 Jun 2017 22:59:59 GMT

Today, I was going through my draft files on this blog and many of them where .html files exported from Blogger (where I previously hosted this blog). And I wanted to change them to markdown files with .md extension. I could obviously use my script to change extension of multiple files, but I thought, there should be a way to do this within emacs.

So I took a deep dive into Emacs documentation and the end result was this function:

Change file extension in Elisp

  (defun my/change-file-extension ()
      (interactive)
      (let* ((new-extension (read-from-minibuffer "Type the new extension including the dot (.): "))
             (new-file-name (concat (file-name-sans-extension buffer-file-name) new-extension))
             (filename (buffer-file-name)))
        (rename-file filename new-file-name t)
        (rename-buffer (concat (file-name-sans-extension (buffer-name)) new-extension))
        (set-visited-file-name new-file-name)
        (set-buffer-modified-p nil)
        (message (concat "File renamed to " new-file-name))))

The above function will ask the user for the new extension, and then rename the file, buffer and also set the visited file name to the filename with new extension.

And on my Spacemacs, I added this key-binding to that function. “o” is reserved for user defined functions in Spacemacs.

  (spacemacs/set-leader-keys "ofrx" 'my/change-file-extension)

If you know a better way of doing this, please feel free to add a comment. Any other suggestions welcome as well.

References

Setting up Apache Airflow on AWS EC2 instance

Fri, 20 Jan 2017 15:53:45 GMT

Setting up Airflow on AWS Linux was not direct, because of outdated default packages. For example I had trouble using setuid in Upstart config, because AWS Linux AMI came with 0.6.5 version of Upstart.

AMI Version: amzn-ami-hvm-2016.09.1.20161221-x86_64-gp2 (ami-c51e3eb6)

Install gcc, python-devel, and python-setuptools

sudo yum install gcc-c++ python-devel python-setuptools

Upgrade pip

sudo pip install --upgrade pip

Install airflow using pip

sudo /usr/local/bin/pip install airflow[s3, hive, python]

Create User and Group

sudo groupadd airflow
sudo useradd airflow -g airflow
sudo passwd -d airflow

This will create a password less user airflow

Initialize Airflow

su airflow
cd ~
airflow initdb

Test run

su airflow
cd ~
airflow webserver

You should be able to view Airflow ui at port 8080

Upstart Config for Airflow Webserver

Now let’s use upstart to manage Airflow process and respawning

This Amazon Linux AMI comes with Upstart 0.6.5, which is very sad. So setuid and setgid doesnot work.

# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

description "Airflow webserver daemon"

start on runlevel [2345]
stop on runlevel [016]

respawn
respawn limit 5 30

env AIRFLOW_CONFIG=/home/airflow/airflow/airflow.cfg
env AIRFLOW_HOME=/home/airflow/airflow/
export AIRFLOW_CONFIG
export AIRFLOW_HOME

pre-start script
  echo "starting airflow-webserver..." >> /var/log/airflow-webserver.log
  echo $AIRFLOW_HOME >> /var/log/airflow-webserver.log
  echo $AIRFLOW_CONFIG >> /var/log/airflow-webserver.log
end script

# exec su -s /bin/sh -c 'exec "$0" "$@"' username -- /path/to/command [parameters...]
exec su -s /bin/sh -c 'exec "$0" "$@"' airflow  -- /usr/local/bin/airflow webserver  >> /var/log/airflow-webserver.log

pre-stop script
  echo "stopping airflow-webserver" >> /var/log/airflow-webserver.log
end script

You should be able to view airflow-webserver in initctl list

sudo initctl list

Start Airflow with upstart

sudo initctl start airflow-webserver

You can find the process id at /home/airflow/airflow/airflow-webserver.pid

Upstart Config for Airflow Scheduler

# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

description "Airflow scheduler daemon"

start on started networking
stop on (deconfiguring-networking or runlevel [016])

respawn
respawn limit 5 10

env AIRFLOW_CONFIG=/home/airflow/airflow/airflow.cfg
env AIRFLOW_HOME=/home/airflow/airflow/
export AIRFLOW_CONFIG
export AIRFLOW_HOME

# required setting, 0 sets it to unlimited. Scheduler will restart after every X runs
env SCHEDULER_RUNS=5
export SCHEDULER_RUNS

# exec su -s /bin/sh -c 'exec "$0" "$@"' username -- /path/to/command [parameters...]
exec su -s /bin/sh -c 'exec "$0" "$@"' airflow  -- /usr/local/bin/airflow scheduler -n ${SCHEDULER_RUNS}  >> /var/log/airflow-scheduler.log

Start Airflow Scheduler with upstart

sudo initctl start airflow-scheduler

This should keep Airflow Scheduler running in the background and respawn it in case of failures.

References

AWS SNS error - Invalid parameter

Thu, 12 May 2016 11:31:15 GMT

So I was playing with AWS Lambdas, and wanted to setup SNS notification for that.

aws sns publish --topic-arn arn:aws:sns:eu-west-1:22834709971:my-awesome-sns-topic --subject 'Subject' --message 'Hello, hello!'

However got stuck on the following error - it just says Invalid Parameter

An error occurred (InvalidParameter) when calling the Publish operation: Invalid parameter: TopicArn

Turns out this has to do with my region settings.

I’d created SNS in eu-west-1 region (Ireland). But default region on my account was us-west-1 So, clearly that is a conflict.

Solution is easy, I just had to set region in ~/.aws/config

[default]
output = text
region = eu-west-1

That’s it!

Game Of Life (in Scala)

Wed, 22 Jul 2015 20:29:26 GMT

I recently came across an interesting problem/game called Game Of Life. A visual version of this is available here. Also, here is a beginners introduction to Game Of Life, or simply Life, written by Alex Bellos at Guardian.

It is a cellular automation problem created by John Horton Conway, a British mathematician in 1970, not a game you play to win or lose. This problem has a few rules that we need to follow.

Rules

A cell in the grid can be dead or alive.
A live cell with less than 2 live neighbours will die.
A live cell with 2 or 3 live neighbours will stay alive.
A live cell with more than 3 live neighbours will die.
A dead cell with exactly 3 neighbours will become alive.

These rules are pretty simple. Once we know the number of live neighbours of a given cell, we can determine it’s fate in a step. In this post I’m going to dicuss method to find neighbours of a given cell in a matrix.

In Game of Life, we take a seed grid and eventually get another grid as a output, by applying the rules to each cell.

Game Of Life grid

Game of Life is played in an infinite two dimensional grid. But for the sake of simplicity let us take a 4x4 grid.

	0	1	2	3
0	Cell(1)	Cell(1)	Cell(1)	Cell(1)
1	Cell(1)	Cell(0)	Cell(1)	Cell(0)
2	Cell(0)	Cell(0)	Cell(0)	Cell(0)
3	Cell(1)	Cell(0)	Cell(1)	Cell(0)

1 is alive and 0 is dead

A cell can have a maximum of 8 neighbours depending on it’s position in the grid, horizontally, vertically, or diagonally adjacent to it.

After applying rules first step, we will get the following

	0	1	2	3
0	Cell(1)	Cell(0)	Cell(1)	Cell(1)
1	Cell(1)	Cell(0)	Cell(1)	Cell(1)
2	Cell(1)	Cell(0)	Cell(1)	Cell(1)
3	Cell(0)	Cell(1)	Cell(0)	Cell(0)

Representation in Scala

Let’s first represent the above seed grid in Scala.

case class Cell(value: Int)

case class Point(row: Int, col: Int) {
  def unapply() = (this.row, this.col)
}

val alive = Cell(0)
val dead = Cell(1)

val row1 = Array(alive, alive, alive, alive)
val row2 = Array(alive, dead, alive, dead)
val row3 = Array(dead, dead, dead, dead)
val row4 = Array(alive, dead, alive, dead)

val board = Array(row1, row2, row3, row4)

Now we have a basic Representation of our models required in Game of Life.

Find neighbours of a given Cell in two dimensional array

Given the co-ordinates of a cell, we should be able to find list of it’s live neighbours.

def findNeighbours(board: Array[Array[Cell]], point: Point):List[Cell] = ???

def findLiveNeighbours(board: Array[Array[Cell]], point:Point):List[Cell] =
        findNeighbours(board, point).filter(c => c.value == 1)

// one step
def step(board: Array[Array[Cell]]): Array[Array[Cell]] = {
  val rows = board.length
  val cols = board(0).length

  // create a new Matrix of same dimensions
  val output = Array.ofDim[Cell](rows, cols)

  // loop through the array and set value for output cell
  // based on number of live neighbours
  for(i <- 0 until rows) {
    for(j <- 0 until cols) {
      val point = Point(i,j)
      val liveNeighbours = findLiveNeighbours(board, point).length
      // apply rules based on number of live neighbours
      output(i)(j) = deadOrAlive(liveNeighbours)
    }
  }
  output
}

// apply rules based on number of live neighbours
def deadOrAlive(liveNeighbours: Int): Cell = {
      if (liveNeighbours < 2) dead
      else if (liveNeighbours == 2 || liveNeighbours == 3) alive
      else if(liveNeighbours > 3) dead
      else if(liveNeighbours == 3) alive
      else dead
}

Find neighbours

Now this is the important bit.

A neighbour is any cell that is adjacent to a given cell, horizontally, vertically or diagonally. (1,1) is adjacent to (0,0), (0,1), (0,2), (1,0), (1,2), (2,0), (2,1), (2,2).

We will implement findNeighbours functions.

def findNeighbours(board: Array[Array[Cell]], point(x:Int, y:Int)):List[Cell] = ???

Algorithm

Here is an algorithm, I found on StackOverflow, by Seb

https://stackoverflow.com/questions/652106/finding-neighbours-in-a-two-dimensional-array

row_limit = count(array);
if(row_limit > 0){
  column_limit = count(array[0]);
  for(x = max(0, i-1); x <= min(i+1, row_limit); x++){
    for(y = max(0, j-1); y <= min(j+1, column_limit); y++){
      if(x != i || y != j){
        print array[x][y];
      }
    }
  }
}

Here is a direct implementation of this algorithm in Scala.

def findNeighbours(board: Array[Array[Cell]], point: Point): List[Cell] = {
  val (row, col) = point unapply
  var cells: List[Cell] = List.empty[Cell]
  val rowLimit = board.length - 1
  val columnLimit = board(0).length - 1

  for (x <- math.max(0, i - 1) to math.min(row + 1, rowLimit)) {
    for (y <- math.max(0, j - 1) to math.min(col + 1, columnLimit)) {
      if (x != i || y != j) {
        cells = cells :+ board(x)(y)
      }
    }
  }
  cells
}

A pattern to use RequireJS config in a separate file

Tue, 23 Dec 2014 11:48:47 GMT

If you have used RequireJS, you most definitely have come across this: Repeatation of requirejs configuration.

You end up repeating the shims and paths in multiple locations. We can minimise this by using requirejs.config(config) method.

requirejs.config() will help you define or override your dependencies configuration.

requirejs.config({'baseUrl': '/my/app/url/'});

Here is an examlpe on how I use requirejs.config method for minimising repeatation.

First let us define the reusable configuration file. Lets name it rconfig.js.

/** Put common requirejs configurations here **/
define([''], function () {
    'use strict';

    return {
        packages: [
            {
                name: 'common-directives',
                location: '/apps/common/directives/'
            },
            {
                name: 'common-filters',
                location: '/apps/common/filters'
            },
            {
                name: 'common-interceptors',
                location: '/apps/common/interceptors'
            }
        ],
        shim: {
            'angular': {
                deps: ['jquery'],
                exports: 'angular'
            },
            'angular-cookies': ['angular'],
            'ui-router': ['angular'],
            'ui-utils': ['angular'],
            'ui-utils-ieshiv': ['angular'],
            'bootstrap': ['jquery'],
            'ui-bootstrap-tpls': ['angular'],
            'ui-bootstrap': ['angular', 'ui-bootstrap-tpls'],
        },
        paths: {
            'jquery': ['/lib/jquery-2.1.1.min'],
            'angular': ['/lib/angular.min'],
            'angular-cookies': ['/lib/angular-cookies'],
            'ui-router': ['/lib/angular-ui/angular-ui-router.min'],
            'ui-utils': ['/lib/angular-ui/ui-utils.min'],
            'ui-utils-ieshiv': ['/lib/angular-ui/ui-utils-ieshiv.min'],
            'bootstrap': ['/lib/bootstrap.min'],
            'ui-bootstrap': ['/lib/angular-ui/ui-bootstrap.min'],
            'ui-bootstrap-tpls': ['/lib/angular-ui/ui-bootstrap-tpls.min'],
            'lodash': ['/lib/lodash.min'],
            'constants': ['/apps/common/constants']
        }
    }
});

Here we have defined the paths for all the common dependencies.

Now let us use it in two different places.

(function (requirejs) {
    'use strict';

    // -- RequireJS config --
    requirejs.config({
        // Packages = top-level folders; loads a contained file named 'main.js"
        packages: ['home', 'dashboard',
        {
            'name': 'dashboard-settings',
            'location': '/apps/dashboard/settings'
        }
        ],
        paths: {
            'rconfig': ['/apps/common/rconfig']
        }
    });

    requirejs.onError = function (err) {
        console.log(err);
    };

    // here we are loading rconfig as dependency
    require(['rconfig'], function (rconfig) {
        // update config
        requirejs.config(rconfig);

        // Load the app. This is kept minimal so it doesn't need much updating.
        require(['angular', 'jquery', './home', 'ui-router', 'ui-utils', 'angular-cookies',
            'common-directives', 'common-filters', 'common-interceptors',
            'bootstrap', 'ui-bootstrap', 'ui-bootstrap-tpls'], function (angular, $) {
                angular.bootstrap(document, ['home']);
            });
    });

})(requirejs);

(function (requirejs) {
    'use strict';

    // -- RequireJS config --
    requirejs.config({
        // Packages = top-level folders; loads a contained file named 'main.js"
        packages: ['home', 'dashboard',
        {
            'name': 'dashboard-settings',
            'location': '/apps/dashboard/settings'
        }
        ],
        paths: {
            'rconfig': ['/apps/common/rconfig']
        }
    });

    requirejs.onError = function (err) {
        console.log(err);
    };

    // here we are loading rconfig as dependency
    require(['rconfig'], function (rconfig) {
        // update config
        requirejs.config(rconfig);

        // Load the app. This is kept minimal so it doesn't need much updating.
        require(['angular', 'jquery', './dashboard', 'ui-router', 'ui-utils', 'angular-cookies',
            'common-directives', 'common-filters', 'common-interceptors',
            'bootstrap', 'ui-bootstrap', 'ui-bootstrap-tpls'], function (angular, $) {
                angular.bootstrap(document, ['dashboard']);
            });
    });

})(requirejs);

The file-layout here is like the following.

Requirejs common config file layout

Hope it helps!