A while ago I updated my site to Eleventy.js. This improved my blogging process considerably, as Markdown is quick to write, and especially code samples are easy to copy-paste without any escaping.

I put all my posts into one Github repository. Using Visual Studio Code, I get nice realtime preview, and once I'm done, I just git commit, git push on my computer, and git pull on the server, and regenerate the site with Eleventy.

Now only issue is that one usually finds 5-10 things to change after every post, and while VS Code has great git support built in, even the simple update process gets a bit tedious with commands being run on SSH side. So I started wondering if I could use the Github webhooks to automate the regeneration. Turns out: YES.

Simple Script to Pull Markdown from Github and Regenerate

First component is of course a small shell script to automate the git pull and regeneration. One could do a more elaborate one, but this worked for me server-side:

#!/bin/sh

cd ~/myblog_post # here are the .md files
git pull
cd ~/myblog_eleventy # here's the 11ty site generator
npx @11ty/eleventy --input=. --output=../apps/myblog_static # hosting dir

Node.js Mini-Server for Github Webhooks

Next, I needed to set up a web server that would get the HTTP POST from Github whenever I push changes. Here your configuration will depend on hosting you have, but Opalstack for example has simple installation of a Node.js application. I usually disable the automatic restarting (crontab -e etc.), use ./stop script and run my server manually for a while to see everything works, before restoring the crontab.

If you choose to forego Github webhook security mechanisms, the code is really simple, but in that case, anyone knowing the addess of your server can flood you with fake push requests. So let's take the high road and use this gist to verify Github hooks! I chose to use Polka so I needed to modify the headers part of the code just a bit:

const { exec } = require("child_process");
const polka = require('polka');
const { json } = require('body-parser');
const crypto = require('crypto')

const port = 12345;
const secret = 'ohreally :)';
const sigHeaderName = 'x-hub-signature-256';
const sigHashAlg = 'sha256';

// Middleware to verify Github "signed" POST request
function verifyPostData(req, res, next) {
  console.log('Verifying signature', req.headers[sigHeaderName]);

  if (!req.rawBody) {
    console.log('Request body empty');
    return next('Request body empty');
  }
  
  const sig = Buffer.from(req.headers[sigHeaderName] || '', 'utf8');
  const hmac = crypto.createHmac(sigHashAlg, secret);
  const digest = Buffer.from(sigHashAlg + '=' +
    hmac.update(req.rawBody).digest('hex'), 'utf8');

  if (sig.length !== digest.length || !crypto.timingSafeEqual(digest, sig)) {
    console.log('Got request with invalid body digest');
    return next(`Request body digest (${digest}) did not match
      ${sigHeaderName} (${sig})`);
  }
  
  console.log('Verification done.');
  return next()
}

polka()
  .use(json({ verify: (req, res, buf, encoding) => {
    // Store raw body data in req rawBody variable
    if(buf && buf.length) req.rawBody = buf.toString(encoding || 'utf8');
  }
  }))
  .get('/', (req, res) => { res.end('Hello, polka!'); }) // just for testing
  .post('/myblog', verifyPostData, (req, res) => {
    console.log('Article repo updated, generating site...');
    exec('~/blog_eleventy/gen.sh', (error, stdout, stderr) => {
      if(error) console.log(`error: ${error.message}`);
      if(stderr) console.log(`stderr: ${stderr}`);
      console.log(`stdout: ${stdout}`);
    });
    res.end(`Hello, Github!`);
  })
  .listen(port, () => {
    console.log(`> Running on localhost:${port}`);
  });

Read post

Continuing from my Project Euler Problem 1, here are the next two Euler problems and some reflection!

Project Euler Problem 2

The second problem asks to find the sum of even Fibonacci numbers numbers below four million. For procedural languages, this is quite walk in the park, but for me a concise method to produce the series in Clojure proved a bit of a brain nugget!

Python 3

For Python, I decided to make a simple generator that yields the Fibonacci numbers below a given treshold N. Rest is just list comprehension:

def fib(N):
    i, j = 0, 1
    while j < N:
        yield(j)
        i, j = j, i+j

print(sum(i for i in fib(4e6) if i%2==0))

C++

In C++ it's natural to just loop with a conditional addition. This time I decided to drop using namespace std; to save a few characters. Without i, j = j, i+j syntax, the temporary variable ij is a bit ugly.

#include <iostream>

int main() {
    int i=0, j=1, s=0;

    while(j<4000000) {
        if(j%2==0) s += j;
        int ij = i + j; i = j; j = ij;
    }
    std::cout << s << std::endl;
}
}

Javascript

Here I could go either way, towards the Python list comprehension or C++ loop. Since generators are part of JS, I chose to try them out!

function *fib(N) {
    let i=0, j=1;
    while(j < N) {
        yield j;
        [i, j] = [j, i+j];
    }
}

console.log([...fib(4e6)].reduce((a,b) => a + b*(1-b%2), 0));

I combined the sum and conditional using reduce. Note that b*(1-b%2) evaluates to zero for off values of b!

Clojure

In Clojure, I wanted really to avoid just looping with while to get a feel of the different ways to generate the Fibonacci numbers. There are many ways to do it, including recursion (which gets really slow unoptimized) and lazy-cat solution that really eluded comprehension, until I realized that with lazy sequences, you can do (map + s (rest s)) and items are pulled from the linked list just in time to construct the next item.

Read post

I've long thought about learning Lisp. Some time ago I ran across Clojure and the map, set and vector implementations felt like a modern addition. It's running on top of JVM which is nice in a sense, but a major pain in the ass when installing. Well, I swallowed my wow to never install JDK again and got Clojure running quite painlessly.

To learn the ropes, I thought it would be a fun idea to rewrite Project Euler problems 1-10 with Clojure to see how it would compare against my language of choice for these types of algorithmic problems, Python.

Just to make it more fun, I decided to implement them in my second favorite language, Javascript as well. Since ECMAScript 9, it's actually a pretty powerful and concise language, and coupling it with Node.js unleashes some serious usage options outside just web pages. Npm dependency hell sucks, but that's a topic for another post. For more bare metal, I'm also doing C++ (basically sticking to C functionality until maps and sets make themselves useful). Let's see how the languages stack up!

Project Euler Problem 1

The first problem in Project Euler is simple, add up numbers below 1000 that are divisible with either 3 or 5 (or both). Modulo operator % can be used to get the remainder of a number when divided by 3 and 5, and basically we add all numbers that have either modulo as zero.

Python 3

With Python, the most concise implementation is to use the sum(iterable) and a list comprehension, netting us a nice one-liner:

print(sum(i for i in range(1000) if i%3==0 or i%5==0))

Read post

After Angular, React.js and Vue, svelte is the new and cool kid on the block. I have to admit, the compile-to-plain-javascript philosophy is nice, and what is especially cool is the fact that with Rollup.js you can easily bundle a svelte app into easy-to-embed component.

I wanted to make a simple embeddable Svelte component for a friend of mine, that would be as simple as possible to include in a blog post, web site or similar with just:

<script src="https://mydomain.com/myscript.js"></script>

The script should ideally just put the component in the same place. Turns out this is really easy with Svelte. Let's dive in!

Setting up Svelte

You may want to read more about rollup-plugin-svelte and Making an app from excellent Svelte tutorial to get more familiar with the tools used here, as I'll only cover the bare essentials.

First you need to install the necessary plugins. We'll start with Rollup and install it globally so you can just use rollup command to do things. If you want, you can also install rollup locally.

user@server:~$ sudo npm install --global rollup
user@server:~$ rollup -v
rollup v2.40.0

I initially got an error saying something about imports and after a bit of googling found out that my Ubuntu version of node was very old (node -v outputted 8.x.x when 10+ was needed) – if you encounter the same issue, upgrade to a newer node (nvm is a great too to do this).

Next make a product directory and install svelte and rollup-plugin-svelte:

user@server:~$ mkdir embed-svelte
user@server:~$ cd embed-svelte/
user@server:~/embed-svelte$ npm init

[answering to npm prompts]

user@server:~/embed-svelte$ npm install --save-dev svelte
user@server:~/embed-svelte$ npm install --save-dev rollup-plugin-svelte
user@server:~/embed-svelte$ npm install --save-dev @rollup/plugin-node-resolve

Simple Svelte component

We'll go really barebones with this tutorial, just a simple heading and a paragraph. Create a Svelte component Embed.svelte:

<script>
export let name; // from props
</script>

<h1>Hello, {name}!</h1>

<p>This component was brought to you by <a href="https://codeandlife.com/">Code
  &amp; Life</a>.</p>

Then we'll need a simple .js script to import that our mini-component and inject it to the DOM of whatever page has included the resulting script. Let's call this embed.js:

Read post

So, I started writing a file database and toolset called fileson to take advantage of AWS S3 Glacier Deep Archive (let's just call it GDA from now on). With 1 €/mo/TB storage cost, it is essentially a dirt cheap option to store very infrequently accessed data like offsite backups.

Why not just use rclone? Well, I disliked the fact that all tools do a ton of (paid) queries against S3 when syncing. I thought a simple JSON file database should work to keep track what to copy and delete. Well, that work is progressing, but as a part of that...

Encrypting on the fly with Python and Pycrypto(dome)

I started thinking that client side encryption would be useful as well. AES is tried and tested, and it's easy to find sample code to do it. But it seems wasteful to first create encrypted files on your hard drive, then upload them to AWS and finally delete everything.

Luckily, the Python AWS SDK boto3 has a great example on how to upload a file to S3 with upload_fileobj that accepts "a readable file-like object". What does that mean? Let's find out!

(note that you need to have the boto3 and pycryptodome libraries installed to successfully run these examples)

#!/usr/bin/env python3
import hashlib, os, boto3

class FileLike:
    def __init__(self, filename, mode):
        self.fp = open(filename, mode)
    def write(self, data):
        print('write', len(data), 'bytes')
        return self.fp.write(data)
    def read(self, size=-1):
        print('read', size, 'bytes')
        return self.fp.read(size)
    def tell(self):
        print('tell =', self.fp.tell())
        return self.fp.tell()
    def seek(self, offset, whence=0):
        print('seek', offset, whence)
        self.fp.seek(offset, whence)
    def close(self):
        print('close')
        self.fp.close()

s3 = boto3.client('s3')

fp = FileLike('hash.py', 'rb')
print('Uploading...')
s3.upload_fileobj(fp, 'mybucket', 'please_remove.txt')
print('Done...')

The FileLike class is a dummy wrapper around basic file functions that prints out what is happening when s3.upload_fileobj uses the provided object.

user@server:~$ ./s3_test.py
Uploading...
seek 0 1
tell = 0
seek 0 2
tell = 357
seek 0 0
tell = 0
tell = 0
read 357 bytes
read 0 bytes
seek 0 0
read 357 bytes
read 0 bytes
close
Done...

So what happens? upload_fileobj seems to:

Read post

I was cleaning up my code folder today and came across my JUnzip library. I realized that I added zipping to the "popular" (40 stars makes it my second most starred repo at the moment) library.

It supports "deflate" method is zlib is present, and "store" if not. You can take a look at the zipping demo code to take a deeper dive.

Additional thanks to Björn Samuelsson for his compact CRC32 routines that were used in case there's no zlib.

Read post