MDWorld

Polymer 2 and Redux

M.D. van Es — Invalid Date

The coming time I will be spending a lot of time developing in Polymer 2.0, so I've done some research and made a small project to test some aspects of setting up a real-world application with it. It is in this repo. It contains two basic elements (polygram-searchbox and polygram-details) that share state via Redux and also I've separated the UI from the API with a container element (polygram-ui-details vs polygram-details). This separation of concerns simplifies reuse (I reuse polygram-ui-details in polygram-marvel-details).

Polymer CLI provides convenient utilities (project scaffolding, building), but the linter and WCT (web component tester) are lacking because they are too limited compared to e.g. ESLint and Karma and in fact it is encouraged to use an separate linter. For that reason, I would have preferred that the Polymer linter would have been provided as an ESLint plugin.

In the examples I use ES6 syntax without a transpiler, but I will soon have to investigate if bundling can be enriched with transpilation because I would at least like to try to use TypeScript. Of course we're losing the module imports (import x from './x') in favor of html imports of elements (<link rel="import" href="x.html">). I think it's a loss, we finally have a standard for imports, why not stick to it?

I would also still like to look at the possibility of adding Redux Thunk and/or RxJS for async event handling. It seems that at least certain features of RxJS are already available in Polymer, e.g. observers on properties and the debounce attribute on iron-ajax.

On a side track I've also done a small comparative case between Polymer and native web components, currently on version 1. This example is in the same repo, in the HelloWorld demo. Polymer builds on web components and implements of component driven development

So far the syntax is very similar, but Polymer is opinionated on certain design decisions (the HTML format, for instance) and adds some convience functions, and more importantly polyfills for wider support (see webcomponents-lite) and a catalog of reusable web components.

Dop SSD update

M.D. van Es — Invalid Date

30-2-2015

I'm replacing the 128GB SSD (Samsung 840 pro) by a 500GB SSD (Samsung 850 EVO) so I have enough space to move the /home partition to the SSD as well. I also had the problem that suddenly my SSD is no longer booting, without using super grub disk, so I might solve this as well.

Currently, I have the SSD with the boot and root partitions on it, a 320GB HDD with the /home (ext3) and swap partitions and a 1,5 TB HDD for storage. I will replace the SSD and the 320GB by the 500GB SSD.

Booting a Ubuntu 13.04 live CD already gives some problems. Right after loading the desktop the system freezes. After multiple reboots it works normally. Using gparted to make exact copies of all partitions from the old SSD to the new SSD (just using copy/paste in gparted). I use a GPT partition table instead of msdos.

Now I disconnect the old SSD and reboot with the super grub disk, like I was doing before. This boots as expected. No I try to fix the boot issue.

Making bootable

After BIOS initialization the message "no bootable partition found" is displayed. Using F12 before boot to select the SSD manually doesn't work either, only super grub disk. I try to follow the official Boot Repair instructions.

Following the second option "install Boot-Repair in Ubuntu" I first login to Ubuntu normally (still booting with super grub disk) to install boot-repair:

sudo add-apt-repository ppa:yannubuntu/boot-repair
sudo apt-get update
sudo apt-get install -y boot-repair && boot-repair

But when I try to run it (boot-repair and selecting "recommended repair") I get a message:

Please create a BIOS-Boot partition: (>1MB, unformatted filesystem, bios_grub flag). This can be performed by tools like gparted, then try again.

So, let's follow these detailed instructions!

The current situation is:

/dev/sdc1 fat32       249MiB    label EFI   flags msftdata
/dev/sdc2 ext4        118.98GiB             flags msftdata
/dev/sdc3 linux-swap   15.62GiB

The fat32 partition was copied from the old SSD. I manually removed the fat32 partition and replaced it with an unformatted BIOS-boot partition and then I rebooted as instructed by gparted. Surprisingly, the partition still shows up as a fat32 partition so I select it and choose > format > cleared. Know the partion is listed as "unknown".

Setting the flag "bios_grub" gives this result for /dev/sdc1:

/dev/sdc1 unknown 249MiB, flags bios_grub

I rerun boot-repair > recommended repair, but get another error:

The boot of your PC is in EFI mode, but no EFI partition was deteced. You may want to retry after creating a EFI partiiton (FAT32, 100~250MB, start of disk, boot flag). Do you want to continue.

So I cancel and try to create the EFI partition from a part of the bios-boot partition with gparted:

/dev/sdc4 fat32      244Mib    label EFI   flags boot, esp
/dev/sdc1 unknown      5MiB                flags bios_grub
/dev/sdc2 ext4       118.98GiB             flags msftdata
/dev/sdc3 linux-swap  15.62GiB

Now I close gparted and run boot-repair > recommended repair for the third time, just to get this error again:

The boot of your PC is in EFI mode, but no EFI partition was deteced. You may want to retry after creating a EFI partiiton (FAT32, 100~250MB, start of disk, boot flag). Do you want to continue.

Now I see no other choice then to continue and I get a progress window telling me that "repair file systems sdb1 this may require several minutes". That's odd, because sdb1 is not the new ssd. Anyway, after 30 minutes of running, I cancelled it and rebooted. And of course it still doesn't work.

Apparently this was all futile and I switch to the non-graphical instructions to restore grub. Now I'm trying:

sudo grub-install /dev/sdc

This is done instantly. After rebooting and using F12 to select the boot device, I can now select the new SSD partition and boot Ubuntu. Progress!

Unfortunately, booting still requires me to select the correct drive manually. This should be easy, just go into the BIOS and change the boot order. But where the boot order is listed, only 2 devices show up: the 1,5TB HDD and the optical drive. After frantic searching in the BIOS and online, I realize something like this might be mentioned in the manual (it is a Gigabyte Z87-HD3 motherboard), and indeed:

go to BIOS, switch to classic mode
go to tab BIOS features
now below boot order there is something like:
- DVD bbs priorities
- hard disk bbs priorities
click hard disk bbs priorities and here I can change the order.

Now it boots directly into Ubuntu! It's using legacy boot and not EFI, but that's fine.

Moving /home

Copying data from one disk to another is easy, but I want to change the formatting from ext3 to ext4 at the same time and of course it contains /home, so I need to update the mount point.

Use gparted to create new partition on the new SSD:

/dev/sdc5 ext4 330.91GiB

Use rsync to copy from the old to the new partition.
- The syntax is rsync --progress -a source dest
- from /dev/sda3 -> /media/ubuntu/9af1dc04-72d9-43a9-a67c-c11f2118dfa6
- to /dev/sdc5 -> /media/ubuntu/211c3ec1-1314-492d-946e-8cb3d24888a9
- so using rsync --progress -a /media/ubuntu/9af1dc04-72d9-43a9-a67c-c11f2118dfa6 /media/ubuntu/211c3ec1-1314-492d-946e-8cb3d24888a9 (WARNING: this is a minor mistake because it creates a dir 9af1dc04-72d9-43a9-a67c-c11f2118dfa6 on the new drive, see below)
- I also run watch -n30 df --type=ext4 --type=ext3 -BG to have an indication of the progress
I immediately see that plenty of dirs give the error "rsync: open dir: permission denied", so I run the same instruction but with sudo. Now it runs much longer. In a couple of hours everything is copied. Running it again logs nothing copied, so it seems to have been successful.

Now the reference of the /home mount point needs to be fixed

the contents of /media/ubuntu/1ea18a24-56eb-4a89-9f54-f9d547c88e35/etc/fstab list as (only relevant parts):

# / was on /dev/sda1 during installation
UUID=1ea18a24-56eb-4a89-9f54-f9d547c88e35 /       ext4   errors=remount-ro  0  1
# /home was on /dev/sdb3 during installation
UUID=9af1dc04-72d9-43a9-a67c-c11f2118dfa6 /home   ext3   defaults           0  2
# swap was on /dev/sdb2 during installation
UUID=079723eb-16f6-487f-b5bb-588d48ce0d55 none    swap   sw                 0  0

listing all UUID with sudo blkid (only relevant parts):

/dev/sda2: UUID="079723eb-16f6-487f-b5bb-588d48ce0d55" TYPE="swap"
/dev/sda3: UUID="9af1dc04-72d9-43a9-a67c-c11f2118dfa6" SEC_TYPE="ext2" TYPE="ext3"
/dev/sdc2: UUID="1ea18a24-56eb-4a89-9f54-f9d547c88e35" TYPE="ext4"
/dev/sdc3: UUID="fbac86d0-4f93-4d8e-a762-dda9c2a61105" TYPE="swap"
/dev/sdc5: UUID="211c3ec1-1314-492d-946e-8cb3d24888a9" TYPE="ext4"

changed fstab to (only relevant parts:

# / was on /dev/sdc2 after move to new SSD
UUID=1ea18a24-56eb-4a89-9f54-f9d547c88e35 /       ext4   errors=remount-ro 0  1
# /home was on /dev/sdc5 after move to new SSD
UUID=211c3ec1-1314-492d-946e-8cb3d24888a9 /home   ext4   defaults          0  2
# swap was on /dev/sdc3 after move to new SSD
UUID=fbac86d0-4f93-4d8e-a762-dda9c2a61105 none    swap   sw                0  0

after rebooting, login failed. Relogged with live CD and found that the /home was copied to a subdir of the new drive. Copied it to the main dir and rebooted. Login in worked normally and is very fast! (How fast? 30 seconds from pressing the power button to seeing the homepage in Firefox)

And finally, when removing the old HDD from the case, I accidentally broke off a system fan pin from the motherboard. The other system fan pins are too far away and the other system fan already died some time ago. Now the only fan is the CPU fan, but so far it seems to be managing with a coretemp of max. 35°C and a CPU temp of max 26°C.

Micro Frontends In A Nutshell

M.D. van Es — Invalid Date

Micro Frontends In A Nutshell

You may have heard of a Micro Frontends recently and felt it a bit difficult to grasp what exactly it is and if it is something you should get involved in. I'll try to give a summary of the what, why, how, and when of the current state. To provide a high-over summary, I'll also add some recommended reading if you would like to get more details.

Why use Micro Frontends?

Do you have a very large front-end code-base?

And by large, I'm talking about 50+ developers in a dozen of teams or more, probably cross-department, working on the same code-base in some manner. Do you have enterprise-scale continuous integration with e.g. GitLab, Bamboo running so many pipelines that the bottleneck is no longer a matter of adding more pods? Do you employ configuration managers or an Ops department to make sure deployments won't affect each other too much? Do you have considerable codebases in incompatible front-end stacks, e.g. because of a migration from AngularJS to Angular >2?

Then chances are you need to use Micro Frontends. Or actually, you most probably are already using Micro Frontends.

What are Micro Frontends?

How is it possible you would not know you are using Micro Frontends? And why are we just now hearing so much about them? The truth is that although the term is relatively new it actually covers any range of solutions to integrate a collection of smaller frontends into one application.

Similar to Micro Service architectures, Micro Frontends facilitate large codebases by breaking them up into manageable pieces. This means:

Technological stack across Micro Frontends in the same application may differ
A Micro Frontend has a clear and concise purpose, following the SOLID principles
Teams of developers maintain one or more Micro Frontend that are isolated in runtime from the rest of the code

The term is now popularized because of the advent of Module Federation in Webpack 5. Webpack 5 has been released towards the end of 2020, but this new major release is taking some time to be integrated in relevant tooling, e.g. Nx and Ng CLI.

Note that Micro Frontends (MFEs) are sometimes also referred to as Micro Apps.

How do you build Micro Frontends?

Since Micro Frontends as a concept are not new, some solutions that can be classified as Micro Frontends are ancient, considering the speed of development in the frontend ecosystem. Here are some of them, to give an idea of how broad Micro Frontends can be interpreted:

Run several frontend applications on different URLs and cross refer them with plain hyperlinks
Run several frontend applications on different URLs on the same page in iframes
Develop frontend components in separate teams and integrate them at build time to be deployed as a deployment monolith
Use macro Web Components as an abstraction layer for components
Develop frontend components in separate teams and integrate them at runtime with Module Federation

I won't go into all the details about the pros and cons of each of these solutions. Instead I refer you to the recommended reading list below.

Note that solutions can be combined: you can have a deployment monolith (that expects components that are all using the same stack) but wrap components in Web Components to provide an abstraction layer and use different stacks to produce the Web Components. Additionally, you can use Web Components in combination with Module Federation for instance if you are migrating towards Module Federation as a Micro Frontends solution. Consider this schematic representation of a web application:

This could be implemented with different platforms like Angular and React by wrapping them in Web Components:

Module Federation is the newest solution and many libraries are still adapting to it. Last year, Nx 12 released with support for Webpack 5 and Module Federation. See a real working example here https://code-star.github.io/nx-reference-shell/ or its source in https://github.com/code-star/nx-reference.

Because Micro Frontends break up a codebase into smaller, more manageable fragments, they are often mentioned in combination with Monorepo solutions like Nx or yarn/npm workspaces. However, it is perfectly possible to implement Micro Frontends without monorepos!

When to use Micro Frontends?

New technology inspires developers to experiment, but Micro Frontends and with that Module Federation are not worth the upkeep for small to medium applications. All-in solutions like Next or Gatsby are great fits for smaller applications and custom Angular applications, when well organized, scale very well up to enterprise level.

However, no framework inherently supports older versions of itself. So if a big bang migration from AngularJS to Angular or any other framework for that matter, you'll end up with some kind of Micro Frontends solution. Plenty of enterprise codebases currently use some combination of hyperlinks and deployment monoliths.

This could look like a bank that offers a set of public pages (e.g. the general home page, and the landing pages of its departments) referencing each other with hyperlinks and a protected monolith app with many components (e.g. checking account, subscriptions to bank products, investments on one page).

Exploring Module Federation can be worth it if continuous integration is slowed down too much because of the large amounts of tests and compilation of all the involved components. But note that there are other approaches, such as using Nx monorepos with properly set up hierarchy and running only affected tests.

Another reason to use Module Federation can be the need to support multiple frameworks. Compared to Web Components, Module Federation improves the runtime isolation of components while simultaneously reducing isolation of shared dependencies to reduce the overall footprint of the application.

Compare to the diagram for the earlier example using macro Web Components, you can see that lodash, Angular and React are only loaded once, despite being used by multiple isolated components:

Want to know more?

If you want to know more about Micro Frontends, Module Federation or Monorepos, you can contact met at @mdworldNL on Twitter. We have experience with enterprise frontend at all the major banks and many governmental departments in the Netherlands.

When you want more background information as a developer, you can also read the articles provided below.

On Reactive

M.D. van Es — Invalid Date

There are a lot of terms in webdevelopment that have some connection to the word "reactive". I found myself mixing them up, so I want to clarify it here.

For the sake of simplicity I will classify libraries & frameworks as a "library" and anything conceptual as a "paradigm".

Reactive Programming

type: paradigm | url: http://www.reactivemanifesto.org

A modern software architecture that fits the current demand for real-time updates. Quote from the Reactive Manifesto: "We want systems that are Responsive, Resilient, Elastic and Message Driven. ... Systems built as Reactive Systems are more flexible, loosely-coupled and scalable." This is all explained in great detail in the manifesto.

Functional Reactive Programming

type: paradigm | Read more

Functional Reactive Programming (FRP) is a way to do reactive programming using concepts from functional programming, such as map, reduce and filter. Silk explains FRP as "[something that] allows us to declaratively set up bindings between data in different parts of our product that keep in sync automatically", making it a way to handle maintainability issues. In short, updating "reactive" variables will also update their values where used in the DOM. Silk has a reactive jQuery plugin which illustrates this quite elegantly. Taken from their site:

var header = new Reactive("initial string");
$("h1").first().rText(header);

// Setting the reactive value now automatically updates
// the text contents of our header element:
header.set("New Header Text!");

I feel this can be compared with two-way binding in JavaScript MVCs.

Also Jafar Husain from Netflix, with 6 years of FRP experience, has an interesting presentation bringing many of the concepts in this article together.

React

type: library | url: http://facebook.github.io/react/

Short version: Facebook's MVC implementation like Angular is Google's MVC implementation.

Focused on the UI (the "View" in MVC). Used on Facebook.com and Instagram.com. Fast because of the use of a virtual DOM and it can be run server-side too.

The use of JSX is optional. JSX is a syntax for using pseudo HTML in JavaScript, more or less like HTML templating. React offers the JSXTransformer library to enable JSX support in JavaScript.

Ractive.js

type: library | url: http://www.ractivejs.org

Short version: A simpeler MVC than Angular.

The learning curve of Angular is quite high. In contrast Ractive.js is designed to be as simple as possible. It was created at theguardian.com and it uses Mustache templates.

These blog posts really goes in depth about Ractive.js vs React and vs Angular

Reactive Extensions (a.k.a. RxJS)

type: library | url: http://reactive-extensions.github.io/RxJS/

Short version: Observables + LINQ + Schedulers, Library for reactive programming in JS.

Reactive extensions exist for several platforms and RxJs is the implementation for JavaScript. The RxJs is incredibly abstract:

"Using RxJs, developers represent asynchronous data streams with Observables, query asynchronous data streams using LINQ operators, and parameterize the concurrency in the asynchronous data streams using Schedulers."

It is modeled after LINQ, which is part of Microsoft .NET and is used for native data querying.

My understanding is that RxJs goes beyond Promises, because it combines Promises, callbacks, DOM input, web workers (threads in JS), and web sockets.

For clarification I tried to find a compelling use case. In theory, this would be anything with events, no matter if it is drag & drop or AJAX calls. An example from their site is an autocomplete that doesn't flood the backend with request because of the reactive programming paradigm. See the autocomplete example on the RxJS examples pages.

Flapjax

type: language | url: http://www.flapjax-lang.org

Short version: Functional Reactive Programming implementation in JavaScript

It has been around since 2006, and the site looks really outdated. Fortunally, there are many such initiatives at the moment, in different stages of implementation:

Bacon.js The site shows this simple example of a counter implemented with Bacon.js. It interprets the data as a stream of clicks on the up or down button:

var up   = $('#up').asEventStream('click');
var down = $('#down').asEventStream('click');

var counter =
  // map up to 1, down to -1
  up.map(1).merge(down.map(-1))
  // accumulate sum
    .scan(0, function(x,y) { return x + y });

// assign observable value to jQuery property text
counter.assign($('#counter'), 'text');

Functional JavaScript

type: paradigm

Short version: implementation of functional programming in JavaScript

The idea is that functional programming should make larger projects easier to understand. Books have been written on why you should want this. Literally, books. There is also a really neat node based workshop, by timoxley. This workshop uses no external libraries.

LoDash

type: library | url: https://lodash.com

According to the site, it is a drop-in replacement for Underscore. Both are libraries with helpers for functional programming. They add functions for typical FP concepts like e.g. map, reduce, and filter. According to the long list of features "not in Underscore" on the LoDash site, it might be considered the better alternative.

Alternative libraries:

Lazy.js, adding lazy evaluation. Actively under development.
Functional Javascript, last updated in 2007.

Real-time web

type: paradigm

Short version: pushing information from the server to the client

Any combination of libraries, technologies or practices that allow server-side push of information to the browser instead of relying on client-side polling. It's basically the new name for the Comet model of programming. By far the most obvious example is web sockets, which is widely supported (e.g. IE 10+). I made a demo for web sockets implemented in Node recently, but it is based on a earlier implementation in PHP that I did somewhere in March 2013. There are plenty of frameworks that make real-time web easier to develop with:

Firebase, web app framework that allows three-way binding when combined with e.g. Angular. This means it syncs database updates straight to the browser.
Pusher, multi platform server side and client side library providing plenty of documentation for implementing common real-time applications, like push notification or a chat client. Provides standardized fallbacks for web sockets.

Responsive/Adaptive design

type: paradigm | Read more

Short version: has nothing to do with reactive, but it sounds like it...

I feel that I should mention this because the first time I heard about reactive, I assumed it classified as a design method, like responsive or adaptive design. It obviously has nothing to do with design, but to clarify:

responsive design is a set of liquid designs, one for each breakpoint.
adaptive design is a set of static designs, one for each breakpoint. E.g. the current design of mdworld.nl is adaptive. Try changing the window width from 700px to 900px, 1000px, 1300px, and 1700px.

NBI Quickscan app

M.D. van Es — Invalid Date

Handige tool voor financieel adviseurs om op eenvoudige wijze eventuele hiaten bloot te leggen.

In een oogopslag doorrekenen van de gebruikelijke scenario’s (langleven, kortleven, AO WW) op basis van besteedbaar inkomen. Op deze manier kan op een simpele en snelle manier een inzicht gegeven worden van het netto besteedbaar inkomen bij verschillende scenario’s.

Bepalen van verzekeringsbehoefte bij verschillende gebeurtenissen (Woonlastenverzekering, ORV)

Voor vragen of opmerkingen ben ik bereikbaar via Twitter.

Please address any questions or remarks to my Twitter.

This is my first real privately produced Android app, although I've worked on several apps through my employer before.

The app was built with the Ionic framework and contains many Karma unit tests for the calculations. It also provided me with an interesting source of working with Gulp and Cordova hooks.

Elm for front-end developers

M.D. van Es — Invalid Date

Elm for front-end developers - how we dove into Elm to rebuild part of our site.

Why Elm?

The benefits of Elm (no runtime errors, typing) is what makes it accessible to our back-end developers. They are used to the functional style of Scala and in general to a workflow with a compiler.

On the other hand, this workflow is also what makes the learning curve enormously steep to front-end developers; if you are used to the front-end workflow, working with a compiler is very different. Back-end (e.g. Scala) developers iterate by writing code and seeing if it compiles. Because of (unpredictable) runtime errors and intransparent state in conventional front-end projects, front-end developers have another kind of iterative workflow: write a small bit of code, see what the result is in the browser. And this means leaving the IDE, navigating your app and opening your browser devtools.

The popularisation of live reloaders and the emphasis on a fast build chain (linters, precompilers) originates from this workflow. Although ultimately developing in a strongly typed and compiled language will have less errors, it does require a different programming style.

Besides adapting to this workflow, the functionally pure coding style is infamously difficult to grasp for the uninitiated (See Functional Programming Is Hard, That's Why It's Good, also just Google “functional programming is difficult”). You think you are comfortable with functional concepts (e.g. immutable state, pure functions, higher order functions, currying) and operators (e.g. map, filter, reduce) in JavaScript? This will be of very little help in Elm, because it is not possible to combine functional style with an OO style (or to fall back to OO when you are completely lost).

Objects have no meaning in Elm, Elm is a true functional language, like Haskell and Scheme. JavaScript on the other hand is flexible, multi-paradigm (imperative, functional, prototype-based) and scales from development by absolute beginners to enterprise.

Immutability

Take for instance immutability. In JavaScript (without libraries) the meaning of immutability is limited to the const keyword, which prevents you from reassigning to declared variables.

const x = {val: 'foo'};
x = {}; // prevented

It is still possible to change the contents of a constant however, with property reassignment.

x.val = 'bar' // value of x is now {val: 'bar'}

Immutability in Elm is much more invasive: there are no variables, only constants. Assignment can be written inline or take a chained style for clarity:

-- inline
myVar : String
myVar = "Hello, " ++ toString(add 5 ((\x -> x + 1) 4))

-- chained
myVar : String
myVar = 4
    |> (\x -> x + 1)
    |> add 5
    |> toString
    |> String.append "Hello, "

Note that the order is reversed to what you may expect in this chained notation, it reads from bottom to top.

This is similar to chaining (functional) operators without storing intermediate state in JavaScript, e.g.:

const retweetAuthors = listOfTweets
    .filter(tweet => tweet.isRetweet)
    .map(tweet => tweet.author)

Although the order here is from top to bottom.

If a value can be changed (e.g. by user input or server updates) you need to use “update” functions in Elm. These will update the entire or partial application state by creating a new state and updating the part of the model that is relevant for this event. Example:

updateModel : Msg -> Model -> Model
updateModel msg model =
   case msg of
       Name name ->
           { model | name = name }

Which means that in the case the message is “Name”, create a new “model” where model.name is set to the supplied name.

An advantage is that over the course of use of the application a list of states is recorded, which enables time travel debugging (like in this GIF).

This might look familiar if you have used Redux. Redux is partially inspired by Elm and you might recognize the [dispatch, action] cycle in the way Elm updates the model. In that respect, it is is also similar to state stores in RxJS:

const increase = Rx.Observable.fromEvent(increaseButton, 'click')
    // We map to a function that will increase the count
    .map(() => state => Object.assign({}, state, {count: state.count + 1}));

Here you can see that a new state object is created, and only the count property is updated.

Another familiar aspect of Elm is the use of Virtual DOM. Instead of manipulating the DOM directly, an in-memory representation of the UI (the virtual DOM) is modified, and that is synchronized to the real DOM. This improves performance dramatically and is one of the selling points of React.

With all of this in mind, we set out to rewrite our company site in Elm. The current implementation of the site is outdated and has all content stored locally. We want to migrate to a JAMstack approach, with the content stored in cloud services (Youtube, Twitter, and photos in Cloudinary) and to pull the data in through the respective APIs with Elm. The idea is that Elm should be more accessible to our Scala devs, while still providing a mature environment for our front-end devs so maintenance can be done by either.

Of course, all devs try to dive right in and soon are discouraged when this approach does not seem to work here. So maybe let's read the Elm docs. There actually are quite a bit of docs. Maybe first finish the tutorials. Actually now I am even more confused….

Toolchain/workflow

No matter if it is Elm, Polymer, or any other New Solution that does not use ES6 modules, it means you can’t use the tools you are probably used to as a front-end developer, like ESLint. You are now at the mercy of the maturity of the ecosystem of the New Solution. Fortunately, it looks like Elm has a very active community and covers Webpack integration, for instance, right in the documentation.

Elm-reactor and external CSS

When starting out with Elm, you usually use elm-reactor to compile and serve Elm files. This offers a web interface that lets you navigate Elm files and when opening them serves them as HTML. Quite soon, you will find out that this presents a problem when you want to add external global resources, like CSS. The Elm app served by elm-reactor is compiled to JavaScript and wrapped in an HTML. But you can’t specify what that HTML should look like, so you can’t add e.g.

<link rel="stylesheet" href="style.css" />

You can (and eventually will) create your own HTML and build with e.g. webpack, but there is an intermediate solution if you want to get to know Elm better first, while keeping the benefits of elm-reactor, such as live reloading and the time travel debugger. It is not very well documented, but you can create an HTML file that directly loads an Elm file. Elm-reactor will automatically compile and live update it. The HTML would contain:

<script type="text/javascript" 
    src="/_compile/app/src/Main.elm"></script>
<script type="text/javascript">
    runElmProgram();
</script>

We did notice that local custom fonts that were added while serving this way were not loaded. Custom fonts from a CDN will probably work.

Webpack

First I set up a Webpack config conforming to this guide. Before, I loaded an image in my Elm code, but now this image gives a 404 when I use the webpack-dev-server. It is also not copied to /dist when building. Webpack does not seem to resolve the image as an asset, which is a known issue. There are workarounds, like setting the image as a CSS background and have the image dependency resolved by the css-loader, but that does not conform to a11y guidelines for informative images.

The suggested solution is to use elm-assets-loader. Additional to the instructions:

I had to add ../ as a prefix to the image path
It was helpful to run webpack instead of webpack-dev-server, it gave more informative errors
I had to remove this line from the webpack config: noParse: /\.elm$/, because it failed on an SVG import without a compile error, but with a runtime error: "Uncaught ReferenceError: require is not defined"

This last issue arose because it is suggested in elm-webpack-loader to set noParse: /\.elm$/, conflicting with elm-css-webpack-loader, which I added later.

The cause of the error and the workaround to set noParse: /^((?!Stylesheet).)*\.elm.*$/, instead, is explained in elm-css-webpack-loader.

Getting the debugger back

When using Webpack, you don't use elm-reactor so you don’t have the time travel debugger. This seems obvious: there is no Elm code anymore, it has been compiled to JS. It is very difficult to find documentation that mentions this. However, when using elm-webpack-starter it actually runs with the debugger!

It seems this is actually an option for the elm-webpack-loader in the webpack.config.js. First we used this:

'elm-webpack-loader?verbose=true&warn=true'

Which is equivalent to:

{
   loader: 'elm-webpack-loader',
   options: {
       verbose: true,
       warn: true
   }
}

And here the option "debug" can be added:

{
   loader: 'elm-webpack-loader',
   options: {
       verbose: true,
       warn: true,
       debug: true
   }
}

I had trouble finding this option, because I had looked at the elm-webpack-loader readme, but I searched for “debug” in the readme, and there is actually only a reference to all the options in the source.

Unit tests

We set up unit tests (see test/Assets.elm) and modified the npm tasks such that:

Postinstall runs elm package install for / and also for /tests/
The new task npm format-validate validates /app/src and /tests/*.elm (as a quick way of excluding /tests/elm-stuff)
npm test runs format-validate and elm-test
npm run build first runs npm test and then starts webpack-dev-server

Speeding up the build on Travis

After adding a unit test (i.e. elm-test) the build speed is taking over 45 minutes on Travis CI. One suggestion is to cache the elm artifacts.

Add this to travis.yml:

cache:
  directories:
    - elm-stuff/build-artifacts
    - tests/elm-stuff/build-artifacts

The first build after adding this should be still slow, because the build-artifacts will have to be cached, but the second build should be faster:

build #23 (first build with cache on): 46 minutes 48 seconds
build #24: 5 minutes 53 seconds

So it is worth the trouble!

It would be possible to cache elm-stuff (where Elm dependencies are installed) instead of elm-stuff/build-artifacts, but this would require manually clearing the cache when new Elm dependencies are added.

Combining several components in one Elm app

We started out with several proofs of concept, so we have a separate TwitterFeed app and DiceRoller app that need to be integrated into the site app. I’ve followed my refactoring attempt (for splitting up one huge Elm file) that I did in my first Elm experiment, which helped integrating the models and update functions.

The TwitterFeed has an initial Cmd. This must somehow be called by the initial Cmd of the main page app.

This is the init of the Model and Cmd of the main page app. The TwitterFeed model has been added, but the initial Cmd is still "none":

init : ( Model, Cmd Msg )
init =
   ( Model "Elm" Material.model initialDiceRoller TwitterFeed.State.initialModel, Cmd.none )

I want a custom init cmd to be called:

init : ( Model, Cmd Msg )
init =
   ( Model "Elm" Material.model initialDiceRoller TwitterFeed.State.initialModel, initCmd )

On init, I want to call a service to get Twitter messages. In the standalone TwitterFeed app this looks like this:

initTwitterFeedCmd : Cmd TwitterFeedMsg
initTwitterFeedCmd =
   Task.attempt NewTweets fetchTweets

It is not possible to create an initCmd in the main page app that calls initTwitterFeedCmd directly:

initCmd : Cmd Msg
initCmd =
   initTwitterFeedCmd

Because initTwitterFeedCmd is of the type Cmd TwitterFeedMsg. However, after adding TwitterFeedMsg to the union type for Msg, i.e.:

type Msg
   = Name String
   | Mdl (Material.Msg Msg)
   | MsgForDiceRoller DiceRollerMsg
   | MsgForTwitterFeed TwitterFeedMsg

Then it is possible to map from Cmd TwitterFeedMsg to Cmd Msg:

initCmd : Cmd Msg
initCmd =
   initTwitterFeedCmd
       |> Cmd.map MsgForTwitterFeed

Conclusion

In the end we simply do not have a strong enough use case for Elm. There is too little state to manage and interaction between components. The TwitterFeed component does have state for instance, but it is so straight forward that Elm offers no advantage over JavaScript alternatives like RxJS state stores or Redux. In this sense Elm would be a good fit for an advanced web app or component with UI controls or complex logic.

Also the steep learning curve for most developers remains, and although this was anticipated for front-end developers it proves more challenging to Scala devs than we first thought.

Fronteers 2014: CSS

M.D. van Es — Invalid Date

After my visit to Fronteers 2014 I was surprised to see that the proportion of strictly development related talks seemed lower compared to previous years. Although most talks were of the highest quality, there were quite some talks about management, infrastructure, etc. Great talks, just not applicable to my work.

In fact, I was secretly hoping for some new tools (e.g. Traceur) or maybe FRP Javascript. I was introduced to both Sass and Grunt at Fronteers once, and they changed my life!

Also, no talk on CSS by Lea Verou this year, but there was a very interesting talk about CSS Animation by Rachel Nabors. I can recommend watching the presentation, but I don't want to go deeper into CSS animations here.

Icon Font

The talk about the Etsy Infrastructure did give me the idea to build a custom icon font for a client that required a lot of custom icons in their web app. Because these are all flat icons, it was possible to use an icon font (considering that you're basically stuck with 1 color per glyph). Unfortunately, the source has a closed license, but I can give the generic outlines of my approach.

The core is the Grunt module grunt-fontsmith, the documentation is a bit sparse, but in the end I used a simple configuration to convert my directory of SVGs to font files and a JSON file with the mapping from icon name to a unicode character. This unicode character can be used to reference the icon in the CSS, exactly like e.g. Glyphicons does. Although grunt-fontsmith can generate SCSS (and CSS, Less, stylus), I eventually opted not to use it. Instead I wrote a small Node script that converts the JSON directly into an SCSS file with the class names that I need, and also sorts all of the variables and class names alphabetically (which grunt-fontsmith can't do for as far as I know).

Also, I patched grunt-fontsmith to have a significantly higher timeout when generating the fonts. It uses PhantomJS to call the Icomoon font generator in the browser (although I have my reservations about this approach, I don't have the time to do it better, so I'll kindly shut up). In my case this might take up to 15 seconds, which is not a problem, because I only run it when an icon is added. Unfortunately the internal timeout in grunt-fontsmith is lower, so I changed it in grunt-fontsmith/node_modules/fontsmith/node_modules/icomoon-phantomjs/lib/icomoon-phantom.js, in the function waitFor(checkFn, timeout, cb).

Basic CSS

At the end of the talk by Nathan Ford about Markup, he includes two examples of basic CSS that I thought where quite clever to have a visual representation of. It seems to clarify much more than the written documentation. I've elaborated on these examples and made two demos that I will use for teaching and personal reference.

Visual Representation of the CSS feature box-sizing

Chrome, Firefox, and IE8+ all support the feature box-sizing. The value padding-box for this property is only supported in Firefox, so I will leave it out of scope for this demo. Border-box is the boxmodel IE uses in quirks mode.

Below are two divs. Both divs have the same height, width, and padding.

By default (content-box), the padding is not included in the box height/width. Height/width is set on the content of the box. The div is 80 + (2*20)px wide.

When using border-box, the padding and border are included in the box height/width. Height/width is set on the border of the box, and the border is outside the padding. Margin is still added to the box height/width.

.pad-out {
  /* default */
  box-sizing: content-box; 
  height: 80px;
  width: 80px;
  padding: 20px;
}

.pad-in {
  /* exception */
  box-sizing: border-box; 
  height: 80px;
  width: 80px;
  padding: 20px;
}

The checkerboard pattern is a div with 100% height and width. The squares are exactly 20px x 20px. It's purpose is to show the relative dimensions between the two boxes.

Compare also these minimal examples of a div, one with border-box and one with content-box box-sizing.

.pad-simple {
   box-sizing: content-box;
   height: 80px;
   width: 80px;
   padding: 20px;
}

.pad-simple {
   box-sizing: border-box;
   height: 80px;
   width: 80px;
   padding: 20px;
}

Sources:

Visual Representation of the CSS features float and clear

I have often used the CSS properties float and clear, but mostly just for putting two block elements inline. This means I usually stick with float: left and clear: both. Now, I would rather use display: inline-block. But how exactly do floats and clear work? I will show some examples.

For all the following examples, the numbers indicate the order of the blocks in the HTML.

All elements float left, but the green element clears left. Clear both would have the same result, because left is included in both. The blocks are essentially broken off to the next line at the green element.

float: left;
float: left;
float: left; clear:left;

1 2 3

All float left, but green clears none (default). No elements are broken off to the next line, although they can still break off if they exceed the width of the parent element.

1 2 3

float: left;
float: left; clear: none;
float: left;

All float right, but green clears right. Again, clear both has the same effect, because right is included in both.

float: right;
float: right;
float: right; clear: right;

1 2 3

Element 1 and 3 float left. Element 2 floats right. Element 3 explicitly clears none. The other two elements implicitly clear none, because it's the default. Clear none is only to override elements that are set to clear left, right or both.

1 2 3

float: left;
float: right;
float: left; clear: none;

Like the previous example, both element 1 and 3 float left. Element 2 floats right. This time, element 3 clears left.

1 2 3

float: left;
float: right;
float: left; clear: left;

These examples might come in handy for reference. The next post will be a linkdump of interesting subjects/tools from Fronteers 2014.

Discourse on Ubuntu 14.04

M.D. van Es — Invalid Date

I wanted to try and use Docker after hearing such great things about it. Since I might want to roll out a decent forum at some point, I wanted to look into installing Discourse. For the moment, this will be done on my desktop.

I start with the installation instructions for Docker on Ubuntu 14.04.

For Discourse it's needed to follow the instructions for the newest version of Docker and not to use the default Ubuntu package.

I proceed with this tutorial that specifies to run:

wget -qO- https://get.docker.io/ | sh
mkdir /var/docker
git clone https://github.com/discourse/discourse_docker.git /var/docker
cd /var/docker
cp samples/standalone.yml containers/app.yml

I modify /var/docker/containers/app.yml (all relevant changes to app.yml are at the bottom)

DISCOURSE_HOSTNAME: 'localhost'

but on running start sudo ./launcher start app I get:

2014/11/16 10:46:31 Error response from daemon: Cannot start container 270d47cd8401c50f13e0a1e51428f5b9833a9480b53bc98735df55b425a70b6d: Error starting userland proxy: listen tcp 0.0.0.0:80: bind: address already in use

This could be expected, because Apache runs on port 80.

in /var/docker/containers/app.yml there is an entry:

expose:
  - "80:80"   # fwd host port 80   to container port 80 (http)
  - "2222:22" # fwd host port 2222 to container port 22 (ssh)

After using netstat -antp to confirm port 8080 is unused, I change 80:80 to 8080:80 in the app.yml.

Now I rebuild and restart the container:

cd /var/docker
sudo ./launcher rebuild app # this applies the changes
sudo ./launcher start app

When I use the browser to go to http://localhost:8080/, it immediately gives me Discourse page!

Now I notice, when creating the admin account, that the activation mail is not send. Thinking back I did skip over the mail settings in the app.yml.

Here I had to experiment a bit with my mail settings and this turned out in the end what took most of the time for getting Discourse up and running. One thing I learned is that it's not neccesary to stop the app with sudo ./launcher stop app before making changes or rebuilding.

Soon I find that running sudo ./launcher mailtest app gives much more useful information, thanks to this thread.

I configure the app.yml to use the Google smtp server (settings are below), but sudo ./launcher mailtest app still gives an error:

ERROR: python yaml module not installed - run the following and try again:

sudo apt-get install python3-yaml

After installing this package mailtest tells me there is an authentication problem and apparently I need to allow "less secure apps" access to my gmail account. After setting this up at Google accounts, I can send mail with mailtest.

The activate mail link is now send and I can access the admin panel in Discourse.

I have also tried to set up mail with the SMTP settings of my own ISP, but this gave an error that I could not find any solutions to:

Traceback (most recent call last):
  File "/var/docker/scripts/mailtest", line 138, in <module>
    smtp = smtplib.SMTP(smtp_addr, smtp_port, timeout=5)
  File "/usr/lib/python3.4/smtplib.py", line 242, in __init__
    (code, msg) = self.connect(host, port)
  File "/usr/lib/python3.4/smtplib.py", line 321, in connect
    self.sock = self._get_socket(host, port, self.timeout)
  File "/usr/lib/python3.4/smtplib.py", line 292, in _get_socket
    self.source_address)
  File "/usr/lib/python3.4/socket.py", line 509, in create_connection
    raise err
  File "/usr/lib/python3.4/socket.py", line 500, in create_connection
    sock.connect(sa)
OSError: [Errno 113] No route to host

Since I never use my ISP's mail server, it might simply be that I use the wrong SMTP address or something.

In the end, I use these setting in my app.yml (these are just the edited fragments):

expose:
  - "8080:80"  # fwd host port 80   to container port 80 (http)
  - "2222:22"  # fwd host port 2222 to container port 22 (ssh)

  DISCOURSE_DEVELOPER_EMAILS: 'myemail@example.com'

  DISCOURSE_HOSTNAME: 'localhost'

  DISCOURSE_SMTP_ADDRESS: smtp.gmail.com
  DISCOURSE_SMTP_PORT: 587
  DISCOURSE_SMTP_USER_NAME: mygmail@gmail.com
  DISCOURSE_SMTP_PASSWORD: ******

I wonder now, since I'm starting discourse now with ./launcher start app if I should have renamed the containers/app.yml to discourse.yml to be able to start Discourse with ./launcher start discourse? I don't know how difficult (or easy?) it will be to change this afterwards, but I will leave it for another time to figure this out.

Fronteers 2014: A Glossary

M.D. van Es — Invalid Date

As I noted in my previous post about Fronteers 2014 there were not as many talks about usable new techniques or tools as I would have liked. This doesn't mean that no interesting new terms were mentioned. That's why I will supply a list of the subjects that were new to me.

A Spacecraft for All

An interactive documentary about the ISEE-3 spacecraft. A Chrome experiment using WebGL.

BlobAPI

from the lecture by Paul Kinlan

Blobs (Binary Large OBjects) are common in most programming languages but relatively new in JavaScript. The File API is based on Blob and Blob content can be read with a FileReader object.

CSS Animate

from the lecture by Nathan Ford

CSS generator for CSS 3 keyframe animations at http://cssanimate.com/.

Dialog

from the lecture by Paul Kinlan

The experimental Dialog HTML element is a semantic element that can be used for e.g. modal dialogs. It's only supported in Chrome 37+ at the moment. Naming conventions around this common component are very unclear (modal, dialog, "popup" and backdrop, fade, "fog", overlay) so it would make a very useful addition, in my opinion. Check out this jsfiddle (Chrome only).

ES6 Compatibility Table

An overview of supported ES6 features for different platforms (polyfills, browsers, runtimes).

Flexy Boxes

from the lecture by Nathan Ford

CSS generator for flexbox at http://the-echoplex.net/flexyboxes/.

getUserMedia

from the lecture by Swetank Dixit

API for accessing video/audio devices from the browser. See many demos here, e.g. a real-life color-picker. As noted in the talk, using https will allow for the browser to store the permission to access the multimedia devices. A very detailed tutorial can be found on html5rocks. In this tutorial getUserMedia is also combined with CSS filters to apply real-time effects to a video stream.

Intern

from the lecture by Nicholas Gallagher

The Intern is a test framework for JavaScript like Karma and Testem, focused on compatibility.

Istanbul

from the lecture by Kyle Simpson

Istanbul is a JavaScript code coverage tool. Of course, there is a grunt plugin.

JSComplexity

from the lecture by Kyle Simpson

JSComplexity is an online tool for complexity analysis of JavaScript projects. Fortunately, it has a grunt plugin.

Line-mode browser

from the lecture by Nathan Ford

Rebuild by CERN of the original line-mode browser from 1992. See the line-mode browser.

Penthouse

from the lecture by Dave Olson

A node module (with a grunt plugin and an online tool) to generate the critical path of CSS to speed up page rendering. The browser waits with displaying any page until the CSS is loaded or times out eventually, resulting in a blank page. Of course it is always recommendable to have the CSS combined and minified (and gzipped), but if performance is important it might be an option to load it inline (!) to prevent failing on bad connections.

Instead of loading all the CSS inline, you can convert the critical CSS for the first page (the above-the-fold CSS) to inline CSS in the document head and load the full CSS in the bottom of the body, where the JavaScript sources should be as well. Extracting this critical CSS is what Penthouse does.

Phaser

from the lecture by Thomas Palef

Phaser is an open source framework for building browser based games. See also the games that Thomas built.

Plato

from the lecture by Kyle Simpson

Plato is a JavaScript source code visualization, static analysis, and complexity tool. And yes, it has a grunt plugin.

ServiceWorker

from the lecture by Paul Kinlan

A ServiceWorker is a type of JavaScript Worker. A Worker is a solution to run background processes by spawning isolated threads. Service Workers run separately from a web page (just like an OS level service) but still runs in the browser. It can be used to improve the off-line experience of an app. See this introduction and this implementation.

Shadow DOM

from the lecture by Nicholas Gallagher

The Shadow DOM lets you manipulate parts of an HTML document that are normally invisibly encapsulated within another HTML element. An example is styling the controls (play, rewind) of the video tag with CSS. This introduction shows some examples. It's part of webcomponents, allowing you to create custom components and hiding the contents in a shadow DOM. It also enables scoping of styles. I also made an example in this fiddle.

Spritesmith

from the lecture by Dave Olson

Grunt plugin to create a spritesheet. I use Compass' magic sprites to do this in Sass, but it might be useful if you want to use libsass or LESS.

SUIT CSS

from the lecture by Nicholas Gallagher

SUIT CSS is a reliable and testable styling methodology for component-based UI development. It offers a.o. a naming convention, a pre-processor and CSS testing.

SVGInjector

from the lecture by Sara Soueidan

A library for inlining SVGs with client-side JavaScript. This combines the benefits of storing the resource in an external file with the benefit of using CSS (normally only for inline SVGs).

Traceur

from the lecture by Kyle Simpson

The Traceur compiler lets you write in ES6 syntax and compile it back to ES5 so you can use ES6 features in today's browers. According to the ES6 compatibility table it supports 63% of ES6 features at this moment.

Ultimate CSS Gradient Generator

from the lecture by Nathan Ford

CSS generator for CSS 3 linear and radial gradients at http://www.colorzilla.com/gradient-editor/.

Webpack Module Bundler

from the lecture by Nicholas Gallagher

Webpack Module Bundler is an alternative to RequireJS that isn't necessarily optimized for async loading like RequireJS but supports the much simpeler CommonJS syntax.

WebPageTest

from the lecture by Dave Olson

Very powerful tool for testing performance of sites. The Chrome plugin SPOF-O-Matic helps to test how sites react when external CDNs are unavailable.

WebRTC

from the lecture by Swetank Dixit

Peer-to-peer connections in the browser, supported in FF and Chrome at the moment. Using the data channels, it was even possible to create a client-side streaming torrent client.

will-change

from the lecture by Rachel Nabors

The CSS property "will-change" is a new feature (Chrome 36, FF 36) that can inform the browser beforehand that certain CSS animation may be applied to an element, helping with optimization. See the docs or this in-depth article on Sitepoint.

Xibalba

from the lecture by Dominic Szablewski.

A web based 3D shooter by phoboslab.org build with the Impact Game Engine. Play it here. It now even offers WebVR support.

Yeast

from the lecture by Arnout Kazemier

"Tiny but linear growing unique id generator." A function to create better cachebusting variables, now formalized as an NPM package.

Dop Beeping HDD

M.D. van Es — Invalid Date

Recently I replaced the 1,5 TB HDD that is used as an archive in Dop by a 3TB Seagate Barracuda ST3000DM001. Silly choice apparently, because it is known to have some defects. Imagine my horror when it started beeping at certain intervals.

The time in between is large enough to seem random, but it apparently only happens after awaking from hibernate. After looking up some forums this seems to be a problem with Advanced Power Management with Seagate drives.

By disabling APM with sudo hdparm -B 255 /dev/sdb the sound indeed disappeared. This setting is not maintained after reboots, but it should be possible to configure it in the /etc/hdparm.conf e.g. like this:

/dev/sdb {
    apm=255
}

npm workaround

M.D. van Es — Sun, 06 Dec 2015 23:00:00 GMT

EDIT The issue concerning the compilation of npm for ARM seems to have been fixed, which would outdate this post. The latest builds should be available at https://nodejs.org/dist/. For RPi 1 look for the builds with "armv6l", for RPi2B "armv7l", and for RPi3 I assume you would need "arm64".

Although it's relatively easy to install Node.js on a Raspberry Pi, getting npm to work is a bit more challenging.

It seems there is an issue with compiling npm, at least for ARM. As long as this remains unfixed, there is no npm with the node 0.12.x ARM installation.

A very simple workaround is to copy the node_modules from another system and to run the packages (e.g. bunyan, which is supposed to be installed globally) with node. Since installing (globally or not) is not possible, but node is installed, I will call it directly from the node_modules that I copied from another machine.

So instead of using npm start configured in package.json as node app.js | bunyan I would do node app.js | node node_modules/bunyan/bin/bunyan

Or if I would like to read the log file afterwards:

node node_modules/bunyan/bin/bunyan -o short error.log

This would also work for other systems where npm can't be installed or if npm can't be used to install packages. I've used this on a system behind a corporate proxy, although I've found a better way by now.

Node on RPi

Keep in mind that this still requires Node.js to be installed. For a Raspberry Pi running Raspbmc it's possible to use apt-get to install Node, but it won't be a recent version. Instead, I use the ARM build node_latest_armhf.deb from node-arm.

Compiling Node on RPi

I've also tried compiling Node from source on the Raspberry Pi (model B). It should not be very difficult, but it failed in my case, probably because of the issue with npm, mentioned before. To be complete, I will list my steps.

I had already installed gcc earlier with sudo apt-get install build-essential
I looked up the latest version in http://nodejs.org/dist/
At 11-7-2015 this was http://nodejs.org/dist/v0.12.7/node-v0.12.7.tar.gz

# adapted from http://elinux.org/Node.js_on_RPi
wget http://nodejs.org/dist/v0.12.7/node-v0.12.7.tar.gz
tar -xzf node-v0.12.7.tar.gz # duration ca. 1 min
cd node-v0.12.7
./configure # duration ca. 1 min

Here I would really recommend to start a screen session, because compiling takes ca. 2 hours.

sudo apt-get install screen
screen start a session
ctrl + a, d detach session
screen -dRR reattach to any session running

My first attempt to run make ended with this error:

/usr/include/linux/sysinfo.h:8:2: error: unknown type name '__kernel_long_t'`

From this thread I found the solution to add these lines:

#ifndef __kernel_long_t
typedef long     __kernel_long_t;
typedef unsigned long   __kernel_ulong_t;
#endif

to the file /usr/include/linux/sysinfo.h, before the lines:

    #define SI_LOAD_SHIFT   16struct sysinfo {...

The second attempt to run make failed with this error:

/bin/sh: 1: cannot create /home/pi/node-v0.12.7/out/Release/.deps//home/pi/node-v0.12.7/out/Release/obj.target/libuv/deps/uv/src/unix/linux-core.o.d: Permission denied

So in the end I ran clean and make as root:

sudo make clean
sudo make

after some hours it fails with:

/out/Release/obj.target/v8_snapshot/geni/snapshot.cc"
Illegal instruction
make[1]: *** [/home/pi/node-v0.12.7/out/Release/obj.target/v8_snapshot/geni/snapshot.cc] Error 132
make[1]: Leaving directory `/home/pi/node-v0.12.7/out'
make: *** [node] Error 2

This is either the compile issue as described in the beginning of this article, or yet again another issue. Since I found out about the compile issue at this point I've stopped experimenting and resorted to the quick workaround.

Site updated

M.D. van Es — Thu, 04 Oct 2018 22:00:00 GMT

It's always a hard decision to move to a new platform for my personal site, and this time it was sped up by the growing need for https (for SEO) and the release of GatsbyJS v1.

Up until now I was using a monolitical Apache + PHP webserver, but since the 2013 version of the site, a dynamic webserver was no longer needed for the blog, because it was generated statically with Jekyll. Only the "send mail" component was in PHP and I was running some other PHP apps, and hosting some binaries, that have nothing to do with the blog. I've done away with the PHP apps: communication by mail was already almost zero and the other apps were outdated.

Now the blog has been properly containerized, this means future upgrades will be easier too: the blog is its own app with its own scope. When I will need anything dynamic in the future, I will use Serverless functions.

Another good reason to migrate from Jekyll to Gatsby is letting go of Ruby dependencies and migrating to a NodeJS based platform. With Jekyll I started to run into problems with recreating the Jekyll stack with the correct dependencies to build the blog, in case of restoring after an emergency. This might be caused by my lack of knowledge of Ruby, but it meant it was really time to upgrade to a newer version of Jekyll or start with a new theme. And in the mean time, Gatsby 1.0 was released and I really subscribe to the JAMstack architecture.

With the new setup, I can work with a modern and familiar stack e.g. Babel, React, TravisCI. I still believe in static site generation for blogs, and just like in Jekyll, adding a post in Gatsby is just a matter of adding a Markdown file with some frontmatter. But an improvement is that you can also drop React components in the pages dir that will be automatically picked up as pages. And it is even possible to add a React component inside a Markdown post. Try clicking the screenshots below for a simple lightbox feature and see the source for this post.

For more information about the current stack, see the about page.

Cover images

The cover images for each post, e.g. the image you see on top of this post, are procedurally generated with my project procgen-cover. This was inspired by the talk by Kate Compton on JSConf.is 2018.

Old themes

Static versions of the 2 last themes still run here (2013-2018) and here (2010-2013).

Theme 2013-2018

Theme 2010-2013

Theme until 2010

Polymer 2 and TypeScript

M.D. van Es — Mon, 11 Sep 2017 22:00:00 GMT

For reasons beyond my control I'm working with Polymer 2 at the moment. Although the idea of web components is great, the choice for HTML imports that comes with Polymer 2 makes integration into a modern development stack cumbersome, as will become clear soon. Also, HTML imports are not widely supported by browsers and although polyfills exist, only Chrome (surprise!) will have native support for the foreseeable future.

Using TypeScript seems like a good choice, because static typing helps prevent runtime errors. Additionally it would be a good opportunity to try out the Scala-TS-interfaces project by my colleagues, that can generate TypeScript from a Scala domain model. Unfortunately, adding TypeScript to a Polymer 2 development stack proves to be difficult, whereas using it with Polymer 3 seems trivial. Polymer 3 is currently in preview so it is not a viable option for me for the moment, but it will exchange HTML imports for ES6 Modules. This will make integrating it into a modern development stack much easier. An example already exists, by Paolo Ferretti and follows normal conventions for a TypeScript project.

If you're adventurous, don't need any existing Polymer 2 elements and don't need to run production; stop reading here and use Polymer 3. If you need Polymer 2 read on, but be warned that it won't be pretty.

TL;DR to use TypeScript with Polymer 2 use typescript-batch-compiler or even better twc.

Webpack

For this experiment I will use my existing Polygram project and the result will be available in the TypeScript branch.

The first challenge is to use Webpack with Polymer 2. Although not strictly necessary for TypeScript compilation, it would make sense for importing HTML as modules. Fortunately, Rob Dodson himself wrote an article How to use Polymer with Webpack. It even mentions TypeScript! The article introduces the Webpack loader https://github.com/webpack-contrib/polymer-webpack-loader and explains how it extracts the JavaScript from the HTML of Polymer elements and eventually packages everything into one JavaScript file. I was basically able to copy the webpack.config.js and index.ejs from his demo project, place it into Polygram and that would compile. I moved my custom elements from the root of the project to the src dir and I had to modify the paths to the bower_components and it would basically work.

The most important exception is Redux, the redux-mixin.html can't resolve the PolymerRedux.html dependency (in bower_components/polymer-redux/polymer-redux.html). The polymer-webpack-loader should resolve this, but runtime it logs Uncaught ReferenceError: PolymerRedux is not defined. E.g. for src/polygram-app.html, the loader seems to import the HTML elements that are used in the template element, but not the JavaScript variables that are used in the script element.

The PolymerRedux code is distributed as JavaScript wrapped in a script tag in mainly one file, so it would be easy to extract it to a JavaScript file. Or even to import polymer-redux/src/index.js instead of polymer-redux/polymer-redux.html (although index.js is uncompiled and misses external dependencies that are not installed in bower_components because they are development dependencies of polymer-redux). For now, I just comment out the Redux dependencies.

It is already clear now that the result from Webpack will be one huge bundle.js that inlines all JavaScript and HTML dependencies. This means using the PRPL pattern will not be possible in this workflow, nor will it be possible to have standalone Polymer components and the accompanying Polymer demo pages.

Adding ts-loader

Normally, to migrate a Webpack project from JavaScript to TypeScript, it would be enough to add the ts-loader to the Webpack config and to rename the JavaScript files to TypeScript files.

So I started with changing the extension for the bootstrapping index.js and adding .ts as a resolved extension:

// webpack.config.js
...
entry: path.resolve(__dirname, 'src/index.ts'),
...
resolve: {
        extensions: ['.ts', '.js'],
...

And adding this rule:

// webpack.config.js
{
    test: /\.ts?$/,
    use: [
        { loader: 'ts-loader' }
    ]
}

And creating a tsconfig.json:

{
  "compilerOptions": {
    "sourceMap": true,
  }
}

Everything still compiles, but the JavaScript for the Polymer components is embedded in the HTML and therefore ignored by the new rule with the ts-loader.

Adding the ts-loader to the rule for the HTML files breaks compilation:

// webpack.config.js
{
    test: /\.html$/,
    use: [
        { loader: 'babel-loader' },
        { loader: 'ts-loader' }, // <--
        { loader: 'polymer-webpack-loader' }
    ]
},

Even without changing any of the code itself, compilation fails with:

ERROR in ./src/polygram-app.html
Module build failed: Error: Could not find file: '/home/me/polygram/src/polygram-app.html'.

Well, that just doesn't look healthy. I filed a bug and almost 2 months after my report the maintainers closed the issue commenting that the root cause is with Webpack, so I don't see this will be resolved any time soon.

For now, I will try to work around it by extracting the TypeScript code to a separate file.

Workaround for TypeScript compilation in a Polymer element

After removing the ts-loader line from the HTML rule in the webpack.config.js I set out to extract the TypeScript to a separate file so it can be compiled with the rule that matches ts files.

Roughly, the main entry point for the Polymer elements polygram-app.html contains:

// imports
<link rel="import" href="../bower_components/polymer/polymer-element.html">
...
<link rel="import" href="polygram-details.html">
<link rel="import" href="polygram-searchbox.html">

<dom-module id="polygram-app">
    <template>
        <!-- Style -->
        <style include="iron-flex iron-flex-alignment"></style>

        <!-- Markup -->
        <div class="layout vertical">
            ...
        </div>
        ...
    </template>
    <script>
        // Script
        import format from 'date-fns/format';

        class PolygramApp extends Polymer.Element {
            static get is() { return 'polygram-app'; }
            static get properties() {
                return {
                    today: {
                        type: String,
                        value: function() {
                            return format(new Date(), 'MM/DD/YYYY');
                        }
                    }
                }
            }
        }
        window.customElements.define(PolygramApp.is, PolygramApp);
    </script>
</dom-module>

Since I know the import statement in the script tag works, I can use this to my advantage. Lets create a companion TypeScript file for polygram-app.html named PolygramApp.ts.

// PolygramApp.ts
import format from 'date-fns/format';

export default class PolygramApp extends Polymer.Element {
    static get is() { return 'polygram-app'; }
    static get properties() {
        return {
            today: {
                type: String,
                value: function() {
                    return format(new Date(), 'MM/DD/YYYY');
                }
            }
        }
    }
}

It would be possible to import PolygramApp.ts with <script src="PolygramApp.ts"></script, but I like the standard ES6 module structure of PolygramApp.ts without the added responsibility of registering itself to customElements, so I import it like this:

<!-- polygram-app.html -->
...

<dom-module id="polygram-app">
    ...
    <script>
        // Script
        import PolygramApp from './polygramApp';
        window.customElements.define(PolygramApp.is, PolygramApp);
    </script>
</dom-module>

The result is a failed compilation with 3 types of errors. Let's deal with them one by one.

1. Failing accessors

The is and properties getters require a specifically set target ECMAScript version, the compilation error is: error TS1056: Accessors are only available when targeting ECMAScript 5 and higher. It surprises me that the default ES target is ES3, but it's not a problem to use ES5 or even ESNext here, because the babel-loader will transpile it back to ES5.

Adding "target": "ESNext" to compilerOptions in the tsconfig.json fixes this error.

2. Failing Polymer import

Polymer can't be found for the extends. This is the most difficult of these errors to solve, because it is caused by the preferred module architecture of Polymer 2: because HTML imports are used, it is not possible to use import Polymer from '../bower_components/polymer/polymer-element.html' because this polymer-element does not export Polymer as an ES6 module. The webpack-polymer-loader can resolve HTML imports, but using import '../bower_components/polymer/polymer-element.html' results in an error TS2304: Cannot find name 'Polymer'.

For the moment, I'm just removing the extends Polymer.Element from PolygramApps.ts and window.customElements.define(PolygramApp.is, PolygramApp); from polygram-app.html.

3. Failing date-fns import

To be able to continue resolving the compilation errors, I add a log statement to polygram-app.html:

<!-- polygram-app.html -->
...

<dom-module id="polygram-app">
    ...
    <script>
        // Script
        import PolygramApp from './polygramApp';
        console.log(PolygramApp.properties.today.value());
    </script>
</dom-module>

The import of date-fns originally failed in the TypeScript compilation with error TS1192: Module ''date-fns/format'' has no default export. but at this point that has two different behaviors:

The IDE warns TS2307 Cannot find module date-fns
Compilation succeeds, but this error is logged in the browser: Uncaught TypeError: format_1.default is not a function(…)

I first thought that this was caused by missing typings for the date-fns library, so I tried npm install @types/date-fns but this logs that date-fns actually provides typings.

Eventually I was able to fix the Uncaught TypeError by changing the import in PolygramApp.ts from

import format from 'date-fns/format';

import { format } from 'date-fns';

And the IDE warning by adding "moduleResolution": "node" to the compilerOptions in tsconfig.json.

At this point, although nothing is rendered, because of the added log statement the current date is logged to the browser console.

Failing Polymer import, continued

Now the import succeeds and it is clear that the TypeScript compiler correctly processes PolygramApp.ts, it is time to try to fix the import of the Polymer module in PolygramApp.ts.

A possible workaround will be to not try to import HTML imports in the TypeScript file, but instead to supply those dependencies through the HTML that is importing the TypeScript file. To do this, I change the respective files to:

<!-- polygram-app.html -->
...

<dom-module id="polygram-app">
    ...
    <script>
        // Script
        import PolygramAppFactory from './PolygramApp';
        const PolygramApp = PolygramAppFactory.create(Polymer);
        window.customElements.define(PolygramApp.is, PolygramApp);
    </script>
</dom-module>

// PolygramApp.ts
import { format } from 'date-fns';
const label: string = 'Current Date: ';

function create(Polymer) {
    return class PolygramApp extends Polymer.Element {
        static get is() { return 'polygram-app'; }
        static get properties() {
            return {
                today: {
                    type: String,
                    value: function() {
                        return label + format(new Date(), 'YYYY-MM-DD');
                    }
                }
            }
        }
    }
}

export default { create }

Now everything compiles without errors and the custom elements are rendered again!

Note here that I also added a string type to const label to see if typings work.

Re-enabling Redux

Earlier, Redux was disabled to test Webpack. It was failing with the runtime error Uncaught ReferenceError: PolymerRedux is not defined To re-enable it, I convert the polymer-redux/polymer-redux.html from bower_components to a local PolymerRedux.js, by just removing the script tags.

Because redux-mixin.html, action.html, and reducer.html actually are already JavaScript wrapped in script tags, I just convert them to TypeScript files, for example:

// ReduxMixin.ts
import {combineReducers, compose, createStore} from 'redux';
const PolymerRedux = require('exports-loader?PolymerRedux!./PolymerRedux');
...
export const ReduxMixin = PolymerRedux(reduxStore);

To use it in PolygramApp.ts, it can now be imported like a normal ES6 module:

// PolygramApp.ts
import { format } from 'date-fns';
const label: string = 'Current Date: ';
import {ReduxMixin, reduxStore} from './ReduxMixin';

function create(Polymer) {
    return class PolygramApp extends ReduxMixin(Polymer.Element) {
        static get is() { return 'polygram-app'; }

        static get properties() {
            // Added Redux code here
            ...
        }

        ready() {
            // Added Redux code here
            ...
        }

    }
}

export default { create }

After making similar modifications for polygram-searchbox, the Redux events work again as before introducing TypeScript.

Importing a global variable from HTML

At this point PolymerRedux is loaded from a custom PolymerRedux.js that I made in the previous step by removing the <script> tags from the file in bower_components. Although this works, it would be better to use the file in bower_components directly because it will be easier to handle updates to this external package.

Currently I import the custom PolymerRedux.js in state/ReduxMixin.ts with:

const PolymerRedux = require('exports-loader?PolymerRedux!./PolymerRedux');

To load the HTML from the bower_components, I expect to have to use the polymer-webpack-loader to extract the JavaScript from the script tags:

const PolymerRedux = require('exports-loader?PolymerRedux!polymer-webpack-loader!../../bower_components/polymer-redux/dist/polymer-redux.html');

This fails to compile with the message that PolymerRedux is undefined, so I add the debug-loader to investigate what the result of each step looks like:

const PolymerRedux = require('exports-loader?PolymerRedux!polymer-webpack-loader!debug-loader?id=raw!../../bower_components/polymer-redux/dist/polymer-redux.html');

Thanks to debug-loader it is immediately clear that already before going into the polymer-webpack-loader the script tags have been stripped. Just using require without any loaders turns something likes this <script>foo()</script> into foo() and webpack-polymer-loader is not needed in this case. I do think this only works when the file is completely self contained and does not have dependencies with other Polymer HTML files.

This is the final working import:

const PolymerRedux = require('exports-loader?PolymerRedux!../../bower_components/polymer-redux/dist/polymer-redux.html');

Linting

Although there is a polymer-linter, it is advised to use Polymer Linter combined with other linters, and an obvious choice is TSLint.

The way that TSLint is configured with Webpack means that it will only lint TypeScript that is not embedded in HTML:

// webpack.config.js
new TSLintPlugin({
    files: ['./src/**/*.ts'] // So, this requires none of the TS to be inline in HTML?
})

Before I started with this experiment I thought this might be a problem. But now almost all script has been extracted to separate TypeScript files anyway, so this works quite well.

Of course it is also still possible to run TSLint manually for a file, e.g. ./node_modules/.bin/tslint --config tslint.json polygram-marvel-details.ts

It is still required to run polymer lint manually. As far as I know there is no integration for Webpack yet.

Decorators

I want to see if I can use ES decorators, because decorators conceptually fit with the Mixin pattern used in Polymer for e.g. class MyElement extends ReduxMixin(Polymer.Element). It would be tidy if we could write this as a decorator, especially if more mixins would need to be combined:

@ReduxMixin
class MyElement extends Polymer.Element
...

As a test, I just add an example decorator to the class in polygram-details.ts:

    @readonly
    foo() {
        // This works, but prepends a polyfill to the output
        return 'just testing a decorator';
    }

And in the same file, but outside the class, the definition of the decorator:

function readonly(target, key, descriptor) {
    descriptor.writable = false;
    return descriptor;
}

The compiler fails with:

error TS1219: Experimental support for decorators is a feature that is subject to change in a future release. Set the 'experimentalDecorators' option to remove this warning.

This flag can be added to the compilerOptions section of the tsconfig.json: 'experimentalDecorators': true. The compilation now succeeds, but prepends a small polyfill for decorator to the output. Take this into account when using decorator in many files, because it will cause an overhead that might be avoided by using a third party library that is imported globally.

App vs Element

It is one thing to compile Typescript for a Polymer app, but another thing to use TypeScript for reusable Polymer components.

The next step will be to compile the Polymer components in this test project separately. As a result, each converted component should both be loaded into its own demo page and to be composed into a Polymer app.

The demo pages should be accessible by running polymer serve, conform the normal Polymer workflow.

When just running a Webpack build for the current project with ./node_modules/.bin/webpack --config webpack.config.js, it will build a dist dir containing amongst others an index.html and a bundle.js. This is a standalone app, but this would not be a good workflow to distribute a Polymer component because:

the index.html is a complete HTML document, not just a dom-module
the bundle.js is one huge JavaScript blob that contains not only the compiled JavaScript for the component, but also all templates, TypeScript, Webpack and Polymer polyfills and libraries (like lodash in this case)

The polyfills and libraries need to be kept separate, so that they can be loaded once per project instead of once for every component. The bundle.js is already 2.8MB in size (unminified) / 347kB (minified).

Would it be possible to make a Polymer component that uses the <script src="foo.js"> style import and then do a "naive" compilation from foo.ts to foo.js? Let's first make a minimal example where the JavaScript is extracted from an Polymer component:

Using /polygram-details.html (this is the original, that the TypeScript+Webpack version in /src/ was based on) and /demo/polygram-details (already importing /polygram-details)
Do not run webpack, but just polymer serve and test the demo page
Replace <script>... code ...</script> by <script src="polygram-details.js"></script>, extract the JavaScript to polygram-details.js and test the demo page again: this works.

Now to TypeScript:

Rename polygram-detail.js but leave the reference in polygram-details.html to point to the JavaScript version: <script src="polygram-details.js"></script>
The package typescript was already installed as a dependency, so use tsc: ./node_modules/.bin/tsc polygram-details.ts. This gives errors, but does generate code. The resulting code does not run.

The first errors are:

polygram-details.ts(8,31): error TS2304: Cannot find name 'Polymer'.
polygram-details.ts(33,14): error TS2339: Property '_searchResult' does not exist on type 'PolygramDetails'.
polygram-details.ts(35,18): error TS2339: Property '_searchIAUrl' does not exist on type 'PolygramDetails'.
polygram-details.ts(42,18): error TS2339: Property '_searchResult' does not exist on type 'PolygramDetails'.

Adding declare const Polymer: any; fixes these 4 errors. It tells TypeScript a global variable Polymer can be expected.

This leaves the following errors:

polygram-details.ts(9,16): error TS1056: Accessors are only available when targeting ECMAScript 5 and higher.
polygram-details.ts(13,16): error TS1056: Accessors are only available when targeting ECMAScript 5 and higher.

The current compilation seems to ignore the tsconfig.json, because a similar error was solved earlier by adding "target": "ESNext" in the config. The target can be specified with a flag: ./node_modules/.bin/tsc --target ES6 polygram-details.ts. This runs without errors and works in the browser!

Simple compilation and Webpack

This much simpler approach without Webpack seems to provide a more realistic workflow. Can we afford to leave Webpack out entirely? Let's reiterate its purpose:

Webpack transpiles to ES5 with Babel

As mentioned before we don't need Babel for transpilation, the TypeScript compiler can be set to ES6 or ES5.

Webpack provides a development server with hot module reloading

Hot Module Replacement is mainly to ease development, but we can use livereload combined with polyserve instead which would be acceptable for this use case.

Webpack handles module bundling

We can do without ES6 modules or packaging other resources like images as JavaScript modules, because we already have to deal with Polymer Elements as a component platform. We have to distribute the end result as Polymer Elements to be able to add it to the catalog.

Although Polymer 3 will use ES6 modules, a tool is supposed to become available that can migrate from elements from Polymer 2 to Polymer 3 syntax.

Without Webpack we lose the module polyfill that is injected per file, which potentially saves a significant size overhead, whilst staying closer to the concept of Polymer Element development.

Global JavaScript variables from external modules can be made accessible with the declare placeholder, and it is still possible to use import to import from node_modules. However, when module is set to none in the tsconfig.json, the variable will just be put onto the "global" scope. This is not the true global scope, because it is still contained within the Polymer element, so the variable will be on the Polymer Element scope, and should not leak to the actual global scope.

Import should still be used with caution: it will lead to code duplication if 2 Polymer+TypeScript elements import the same dependency. In that case it would be better to import that dependency via HTML import because the Polymer compiler can deduplicate it.

Webpack can also be used to package CSS as modules, but for encapsulating CSS in Polymer the Shadow DOM can be used. This is actually an aspect of web components that is very well executed.

Automatic compilation

Without Webpack, it is unpractical that for every change to a TypeScript file a manual transformation is needed. Following the example in the previous section, each time polygram-details.ts changes, ./node_modules/.bin/tsc --target ES6 polygram-details.ts must be run. Let's try to automate this without using Webpack.

With tsc

First I make a new tsconfig named tsconfig.inline.json for this use case:

{
  "compilerOptions": {
    "sourceMap": true,
    "target": "ES6"
  },
  "include": [
    "*.ts"
  ]
}

To compile run ./node_modules/.bin/tsc -w -p tsconfig.inline.json. The -w flag keeps the process running and watches for changes in the included TypeScript files.

An interesting side-effect occurs. Naturally, each TypeScript file is going to need the declare const Polymer: any; declaration as a workaround for the fact that the Polymer dependency can't be imported (see previous sections). But because we now use -p, the project flag, the compiler expects all files share global scope. And the second file using declare const Polymer: any; will get an error: Cannot redeclare block-scoped variable 'Polymer'. How can we use the project flag, without letting the compiler share the global scope between all TypeScript files?

A workaround would be to create a TypeScript file that just imports/declares all the expected global variables once. This would make the code less transparent at best and it might even create other scoping issues.

Custom compilation

As an alternative let's try to run compilation with an isolated scope for each TypeScript file. This issue explains that this would be possible by supplying a tsconfig.json for each scope. That would be doable for a limited set of scopes that is static over time (e.g. a back-end codebase and a front-end codebase in the same project). However, it makes no sense from a maintenance standpoint for the current project as it would need a tsconfig.json for each Polymer element.

To be complete, I did try this out. First, set up a base tsconfig that can be inherited:

// base.json
{
  "compilerOptions": {
    "sourceMap": true,
    "target": "ESNext",
    "moduleResolution": "node",
    "experimentalDecorators": true
  }
}

Now for each Polymer TypeScript file a tsconfig, e.g.:

// polygram-details.tsconfig.json
{
  "extends": "./base.json",
  "files": [
    "./polygram-details.ts"
  ]
}

It is now possible to compile/watch polygram-details.ts with tsc -w -p polygram-details.tsconfig.json, but it is still not possible to compile/watch multiple tsconfigs at the same time.

In this case it would be better to forget about the watch flag -w altogether and just use npm watch combined with tsc [changedfile]. You can't use a tsconfig.json combined with an input file path for tsc, so all options must be supplied as flags: tsc --target ES6 --sourceMap [changedFile].

I tried combining this compilation one-liner with a watch script, but I could not get this to work with nodemon, npm-watch or watch, so I wrote a small script:

// ts-poly-watch.js, run with: node ts-poly-watch.js
const watch = require('watch');
const path = require('path');
const chalk = require('chalk');
const tsc = require('node-typescript-compiler');

watch.createMonitor(__dirname, { interval: 1 }, function (monitor) {
    console.log(chalk.gray.bgGreen.bold('TS-POLY-WATCH started'));
    monitor.on('changed', function (f, curr, prev) {
        const ext = path.extname(f);
        if(ext === '.ts') {
            console.log(f + ' changed');
            tsc.compile(
                {
                    'target': 'ES6',
                    'sourceMap': true
                },
                f
            );
        }
    });
});

Now it is possible to watch each TypeScript file and compile it with its scope isolated from the other TypeScript files.

twc

With ts-poly-watch.js it looks like we finally have an acceptable working environment. I have extracted the script to its own project typescript-batch-compiler and npm package because there is much room for improvement and it will be easier to use in other projects if it is an npm package.

So are we now done? In fact there is one more thing I want to explore. During the research I ran into twc. This is a compiler for TypeScript Web Components and can be used to compile TypeScript classes to Polymer 2 elements. Although this sounds like it is similar to my typescript-batch-compiler, here are some preliminary findings:

It assumes a TypeScript file as the entrypoint: in my setup I still stay close to the Polymer setup with a Polymer element (so HTML) that includes a script file. With twc the entrypoint is a TypeScript file that imports an HTML template. A great advantage is that this is more like Polymer 3 and also similar to other component driven frameworks like React, Vue and Angular. The disadvantage is of course that the style will be foreign to other Polymer developers.
It's still very experimental. There is no example in the repo or an explanation of how to set up a basic element in the README, but a general approach is outlined on the project wiki and examples of twc in a project can be found in this generator project.

With the aforementioned wiki, I take these steps:

Run twc in a new subdir of the project appropriately named "twc"
Set up a very basic polygram-twc.ts conforming to the style as outlined in the wiki:

// polygram-twc.ts
import { CustomElement } from 'twc/polymer';
import 'bower:polymer/polymer-element.html';

/**
 * `online-state`
 * Lets you select an online state (online or offline) and reflect the change on a host attribute.
 *
 * @customElement
 * @polymer
 * @demo demo/index.html
 */
@CustomElement()
class OnlineState extends Polymer.Element {
    prop1: string = "online-state";

    template() {
        return `
          <style>
            :host {
              display: block;
            }
          </style>
          <h2>Hello [[prop1]]!</h2>
        `;
    }
}

Run tsc --init to create a new tsconfig.json in the twc dir. This turns out to be important. When I re-use my existing tsconfig.json the build fails with Error: Debug Failure.. This seems to be caused by the line "moduleResolution": "node", which is not needed for this compilation.
Add node_modules/twc/types/polymer.decorators.d.ts to the include section of the tsconfig.json, to resolve certain types.
Build in the twc dir by running ../node_modules/.bin/twc polygram-twc.ts.
A polygram-twc.html is created.

I also converted the original polygram-details.html (the one with embedded JavaScript) to this format. See the result in the repo for this experiment. When working on this conversion, some differences with normal web components become apparent:

twc auto-injects the registration of the component: customElements.define(PolygramDetails.is, PolygramDetails);
import './polygram-ui-details'; is converted to <link rel="import" href="./polygram-ui-details.html">
the is getter, i.e. this: static get is() { return 'polygram-details'; } is auto generated from the class name.
The JSDoc is converted to an HTML comment.

This syntax uses plain ES modules and is therefore also closer to Polymer 3. Still there are some differences. Compare the code for polygram-twc.ts but in Polymer 3 syntax:

// PolymerElement is its own module now, instead of a property of the Polymer namespace. Also, bower is no longer used.
import { Element as PolymerElement } from '@polymer/polymer/polymer-element';

// Aside from inline templates, this syntax can be used too:
//import * as view from './app.template.html';

export class OnlineState extends PolymerElement {
    constructor() {
        super();
        // Property must be defined in the constructor, but this might be a difference with TypeScript and not Polymer 3.
        this.prop1 = 'online-state';
    }

    static get template() {
        // I don't know where the <style> element should go.
        return `<h2>Hello [[prop1]]!</h2>`;
        // Or when using an import:
        //return view;
    }
}

Although I can't find any sources, I heard that Polymer 3 would supply an auto converter from (normal) Polymer 2 syntax. You could use that converter on the output of twc, so this should not be a reason to avoid twc.

Final remarks

Unit testing and coverage support when using TypeScript has not been mentioned, but I hope it is clear that it is unchanged from a normal Polymer 2 application when using typescript-batch-compiler. You can just use WCT), because all components are compiled to a state that conforms to a non-TypeScript Polymer 2 situation.

For the Webpack approach it would be an improvement to see why polymer-webpack-loader is not importing Polymer when using import Polymer from '../bower_components/polymer/polymer-element.html' or import '../bower_components/polymer/polymer-element.html'.

It could also be an improvement to add the prettier plugin to promote a consistent coding style. This could be added to TSLint via Webpack, but could also be integrated in the typescript-batch-compiler package.

npm without internet

M.D. van Es — Mon, 06 Jul 2015 22:00:00 GMT

It's easy to run npm to install packages even on a system without internet access, or behind a corporate firewall. First note that npm has proxy settings, so if you have access to the correct credentials, try that first.

Let's assume you're developing a web application and you use Grunt for managing jshint and uglify tasks. This requires the npm package grunt-cli to be installed globally. All the other packages (i.e. jshint and uglify) can be run locally, so it's sufficient to copy node_modules with these packages from another installation.

Assuming you have another system with internet access/no proxy, and you have a working project with all the packages there:

Zip the node_modules in the project dir to e.g. node_modules.zip.
Go to %userprofile%\AppData\Roaming\npm\node_modules\grunt-cli\
Run npm pack this should result in a grunt-cli-x.y.z.tgz
Collect the node_modules.zip, grunt-cli-x.y.z.tgz and Node.js installer and place them on the target system (e.g. by local network or usb drive).
Prepare the project dir on the target system, e.g. from a Git repo on the network or from scratch. The Gruntfile.js and package.json should already be in the root of the project.
Install Node.js on the target system. This should also install npm. Check this with node -v and npm -v.
Unpack the node_modules.zip into the project dir, such that <project>/node_modules contains e.g. grunt-contrib-jshint.
Place grunt-cli-x.y.z.tgz in some temporary dir, e.g. \tmp and run npm i -g grunt-cli-x.y.z.tgz.
Grunt should now be installed globally, try grunt --version.
Now it should be possible to run grunt in the project dir.

If you want to do a lot of offline installations, you might want to +1 this issue. It explains the process of doing several pack and install steps and how this could be simplified by extending the pack feature to also zip all the depencies of a package.

KnockoutJS and ES6

M.D. van Es — Mon, 04 Jul 2016 22:00:00 GMT

In the past year I've made some ES6 projects, for instance HomeRemote, an ES6 flavored React project. My company mainly uses a KnockoutJS based theme and it would be an improvement to introduce ES6 to this theme. The combination of Knockout and ES6 is not very common, so I've summarized my endeavors.

Precompiling with Webpack

Because I want to use ES6 classes and native modules, which are respectively not well and not at all supported cross browser, I will use a transpiler. Because I will want to look into component based development, I choose Webpack, also because it is pluggable and has e.g. Typescript support.

Although I am aware of the advantages of TypeScript and it's adoption should increase with the advent of Angular 2, I will stick with plain ES6 for the moment, to stay as close as possible to the native syntax.

It is not very clear how to install all the needed dependencies for ES6 compilation with Webpack. It does give feedback on missing modules when running, so with some trial and error, this should be the installer command:

npm i --save-dev webpack webpack-dev-server webpack-merge grunt-webpack babel-core babel-loader babel-preset-2015

Because JShint and JSCS do not support ES6 syntax, I replace them with ESlint:

npm i --save-dev grunt-eslint

I integrate Webpack and ESlint into Grunt, this is very straight-forward:

eslint: {
    options: {
        configFile: '.eslintrc'
    },
    dev: [
        '_js/**/*.js'
    ]
},

webpack: {
    options: webpackConfig,
    build: {
        progress: false,
        plugins: []
    }
},

Both Webpack and ESlint changed the syntax for their configuration since the last time I used it, my configs look like this now:

Webpack 1.13.0

var path = require('path');
var merge = require('webpack-merge');
var webpack = require('webpack');

var common = {

    entry: './_js/app.js',

    resolve: {
        extensions: ['', '.js']
    },

    output: {
        path: path.join(__dirname, 'js'),
        filename: 'bundle.js'
    },

    devtool: 'source-map',

    module: {
        loaders: [
            {
                test: /\.js$/,
                loader: 'babel-loader',
                query: {
                    presets: ['es2015']
                }
            }
        ]
    }
};

module.exports = common;

ESlint (grunt-eslint 18.1.0)

{
  "parserOptions": {
    "ecmaVersion": 6,
    "sourceType": "module",
    "ecmaFeatures": {
    }
  },
  "env": {
    "browser": true,
    "es6": true
  },
  "rules": {
    "no-bitwise": 2,
    "curly": 2,
    "eqeqeq": 2,
    "no-use-before-define": 2,
    "new-cap": 2,
    "no-caller": 2,
    "no-undef": 2,
    "no-unused-vars": [2, {"args": "none"}],
    "no-trailing-spaces": 2,
    "strict": [2, "function"],
    "quotes": [2, "single"],
    "complexity": [1, 3]
  }
}

The Entrypoint

The file app.js is the main entrypoint. It imports all the factories and classes as dependencies and registers them to our application API (bqApp). E.g.:

import otherCustomModelFactory from './es6/viewmodels/otherCustomModelFactory';
import otherCustomTemplateFactory from './es6/otherCustomTemplateFactory';

if (bqApp && blueriq) {
    bqApp.modelFactory.register(otherCustomModelFactory);
    bqApp.templateFactory.registerModelHandler(otherCustomTemplateFactory);
}

The registering used to be done in e.g. the customModelFactory file itself, but when using ES6 modules, I started to notice how this requires a dependency to bqApp in the customModelFactory file, while it's only needed for registering the factory. For the sake of high cohesion, I decided to export the finished factory and register it in the app.js instead.

In advance, one of my main concerns was handling existing variables that the application API puts on the global scope, because the API is not yet in ES6 style. Fortunately, in contrast to Typescript it is not needed to have typing definitions, making it trivial to mix ES5 and ES6 libraries in one app. Because ES6 modules are used in the app.js, it is no longer needed to use an IIFE for scoping (the ES6 modules are scoped at the file level anyway) or strict mode declaration (implicit, because strict mode indicates ES5 syntax). This does mean I need an alternative way to reference global variables. E.g. the bqApp variable is put on the global scope by the application API, which I can't change at the moment, instead of being exposed as an importable module. Supposedly, it would be possible to wrap it in a file that converts the global to a module. For the short term I see two options, though:

/* global bqApp:false */ - this works because the var is on the global scope, it's just ESlint who warns for undefined variables.
const bqApp = window.bqApp; - explicitly reassinging to a local variable, which is basically what I used to do when supplying window.bqApp as a parameter to an IIFE, and then declaring bqApp as an argument in that IIFE.

For the moment I will stick with option 1, although I have included an example of both just to be complete.

Viewmodels and the viewmodel factory

Most viewmodels that we use extend from some existing viewmodel that is build into the application API, such as blueriq.models.BaseModel. A very simple example:

(function(ko, blueriq) {
    var FooModel = function(model, context) {
        var self = this;
        blueriq.models.BaseModel.call(self, model, context);
    }
})(window.ko, window.blueriq);

As said, I was concerned about using existing variables from the application API. I had no idea how the ES6 "extend" would convert to ES5, but fortunately it seems to be backwards compatible with the approach above, because this simply works:

/* globals blueriq:false */

// I want to extend blueriq.models.dashboard.PageModel (which is an old-fashioned ES5 class). This works!

class PageModel extends blueriq.models.dashboard.PageModel {
    constructor(model, context) {
        super(model, context);
        console.log('new es6 PageModel', model, context);
    }
}

export default PageModel;

Now PageModel can be imported into the modelFactory, which decides what viewModel to use based on properties of the provided model.

import PageModel from './PageModel';

const otherCustomModelFactory = function otherCustomModelFactory(model, context) {
    if(model.type === 'page') {
        return new PageModel(model, context);
    }
};

export default otherCustomModelFactory;

It has already been shown how otherCustomModelFactory is imported and registered in app.js.

Templates

Creating a templateFactory, which decides what template to use based on properties of the provided model, is nearly identical to the modelFactory:

/* globals blueriq */

const otherCustomTemplateFactory = function otherCustomTemplateFactory(viewModel, context) {
    if(viewModel.type === 'field') {
        return 'themes/myTheme/template/field';
    }
};

export default otherCustomTemplateFactory;

At this moment, the factory just returns the id of the template. The template itself is requested from the filesystem by the koExternalTemplateEngine (this project has been retired for several years now) or with a custom solution that reads the template from a file that contains all templates. A very important improvement that could be made here is using Webpack to resolve the paths to the templates. It should be possible to write a custom loader that retrieves the contents of a template that is being imported, e.g. like this:

/* globals blueriq */
import fieldTemplate from './template/field.html';

const otherCustomTemplateFactory = function otherCustomTemplateFactory(viewModel, context) {
    if(viewModel.type === 'field') {
        return fieldTemplate;
    }
};

export default otherCustomTemplateFactory;

Custom Binding Handlers

Knockout handlers need to be registered to ko.bindingHandlers, e.g. like this:

ko.bindingHandlers.clickTester = {init: function(elem) {
    $(elem).click(function() {
        alert('you clicked it!');
    });
}};

If I would put this in a module, nothing would be exported. I still need to look into how the system would handle this. Probably it will just execute adding the bindinghandler and return nothing, but for the moment I just export the body of the handler and import and register it in our app.js.

In handlers/clickTester.js:

/* globals $:false */

export const clickTester = {
    init: function(elem) {
        $(elem).click(function() {
            alert('You clicked the ES6 KO Handler!');
        });
    }
};

In app.js:

import {clickTester} from './es6/handlers/clickTester';
ko.bindingHandlers.clickTester = clickTester;

Follow up

I still need to improve the registration of handlers, I don't know if I want them all in my app.js or maybe in a separate file. In that case I will need to figure out how to export "nothing" to my app.js, otherwise the import statement might fail.

So far, our product does not use Knockout components. The style of component based development is very similar to React or Angular 2.0 components, although in practice, it might be closer to Angular 1.x directives. It would be interesting to see how it would translate to the ES6 syntax.

Gaining Root Access to Smart Home Gateway

M.D. van Es — Wed, 08 May 2019 22:00:00 GMT

Recently I bought a Risco Smart Home Gateway, and as I expected it is a rebranded off-the-shelf micro computer. When delivered, the HDMI and optical S/PDIF outputs and IR input hinted that it was not just a dedicated smart home gateway. The OS seems to be loaded from a micro SD card, comparable to e.g. a Raspberry Pi. Also, the z-wave controller is just a USB dongle and not built-in.

After configuring, its network interface shows up on my router as SolidRun. A quick search finds the SolidRun CuBox-i, most probably the CuBox i2eX or CuBox i4p, because it also has an eSata port.

These range from $100 - $150. Not too bad, considering the Risco Smart Home Gateway is around €300. Of course, this includes a z-wave adapter (around €50) and their special software. Their software is actually much less flexible than e.g. Domoticz, but I am sure it makes up for that in security and robustness.

Solid Run has a community site for developers with plenty of information.

The CuBox is an ARM micro computer with an FreeScale i.MX6 SoC and that means it can run several flavors of Linux, Kodi and in theory and old version of Android. An in-depth review can be found here.

I won't judge the performance or review it further, but it might be worth mentioning that by know it is at least superseded by the Raspberry Pi 4, because it can play h265 video.

Now if you are bored with your Risco Smart Home you could just install a different image on the micro SD card and use it as a media player or something, but let's see if we can gain more control over it as is.

You could use Wireshark to intercept messages between the Risco Smart Home device and the alarm system, but first I want to try to SSH in and/or inspect the software it is running.

In this post, I will display the IP as 192.168.0.xxx. Accessing that IP in the browser shows nothing running on port 80, but the IP is pingable.

nmap -v -sT 192.168.0.xxx on the IP shows:

PORT     STATE SERVICE
22/tcp   open  ssh
8080/tcp open  http-proxy
8443/tcp open  https-alt

Opening http://192.168.0.xxx:8080 in the browser it redirects to http://192.168.0.xxx:8443/static/index.html:

HTTP ERROR 404

Problem accessing /static/index.html. Reason:

    Not Found

Powered by Jetty://

Opening https://192.168.0.xxx:8443 in the browser gives the same as 8080, but with SSL. It uses a self-signed certificate by openHAB. This is open-source home automation software and this may be useful if we gain access to the device!

It is possible to create an SSH connection, but it requests a password. Trying some default root passwords failed. Now we know the certificate is from openHAB, maybe the CuBox is running openHAB? The default credentials openhabian/openhabian do not work.

It may be possible to inspect the micro SD to get more information. When disconnecting the gateway from power/utp the alarm does not go off (good!), and the micro SD can be inspected on a different machine:

The micro sd has several partitions: boot, postgre, rootfs, and scratch.

rootfs: there is a /home dir, but it is empty. there is a /linuxrc dir that seems to point at bin/busybox. This seems to be a BusyBox installation.

scratch: contains some ssh certificates. Copied to local disk.

After reconnecting the gateway to power and UTP the message "Communication with the Gateway failed." is shown in Risco cloud. After a couple of minutes it works fine. However, the button controller that was paired (that did not work) is not showing up anymore. The Fibaro plug does show up and still works as expected.

It is not possible to login with default BusyBox credentials, so after disconnecting again, taking a closer look at the volume scratch: it contains a file @ssh/sshd_config with a line PermitRootLogin yes.

So assuming this is the SSH config that is used runtime, we need to somehow find the root password, and then we can SSH with ssh root@192.168.0.xxx

Following this suggestion:

see the /etc/shadow or /etc/passwd (if the /etc/shadow not exist) with texteditor

I found that, indeed, /etc/shadow contains (values are fake):

root:$1$foo$bar:10933:0:99999:7:::

So there is in fact a root entry with a hashed password visible. One option would be to update the hash string with a hash string of a password that we already know, a method I used years ago when adding users for a deprecated application by directly inserting lines into a database table. However, I have no idea what effect this would have on an OS, so I would prefer to refrain from modifying the filesystem at this point. Additionally, assuming that the provided SD card image always uses the same root password, it would be valuable to know the password to be able to log into other devices.

Another option would be to try to retrieve the password from the root entry in /etc/shadow. More information about why passwords are stored in /etc/shadow instead of /etc/passwd here.

From this article I also learned how to read the shadow file: the password entry ($1$foo$bar) is in fact three properties separated by $. The $1 indicates MD5 hashing was used, which is notoriously easy to crack. The second part ("foo") is the salt and the third part ("bar") is the hash from the password + salt.

This article also shows that using the hash calculated from the salt and the password with openssl. This is great, because know we can write a small script to try out many passwords without trying to login to SSH and being kicked out for too many failed responses. The command looks like this, with -1 indicating md5, foo as the salt and secret the actual password.

openssl passwd -1 -salt foo secret # output: $1$foo$ygQElgzdVoe51UiGBN5TV/

It is easy to do this in a bash script, but of course there already exists some open source tool: John the Ripper (info on usage)

First run sudo unshadow rootfs/etc/passwd rootfs/etc/shadow > mypasswd.txt where unshadow is a util from john.
Now remove all the entries from mypasswd.txt except the root user.
Now run john mypasswd.txt --fork=4.

The fork flag enables multi threading, in this case on 4 threads. While the command is running, I see that 4 out of 8 cores have a 100% load. After approximately 7 hours a password is found. Please note, that the other threads keep running, but a nice overview is giving of how long the processes have been running and at what passwords they are trying.

Logging in to SSH as root works! (So it was indeed the correct sshd_config). A prompt is shown with only "#". Running df -h gives:

Filesystem                Size      Used Available Use% Mounted on
/dev/root                 2.8M      2.3M    444.0K  84% /mnt/initrd
devtmpfs                239.2M         0    239.2M   0% /dev
/dev/disk/by-label/rootfs
                          1.0G    284.2M    544.7M  34% /
tmpfs                   249.4M      4.0K    249.4M   0% /dev/shm
tmpfs                   249.4M      9.1M    240.2M   4% /tmp
tmpfs                   249.4M    120.0K    249.2M   0% /run
/dev/mmcblk0p1           63.0M      5.9M     57.1M   9% /boot
/dev/mmcblk0p3            1.0G     93.7M    828.9M  10% /var/lib/pgsql
/dev/mmcblk0p4            1.0G     23.7M    898.7M   3% /var/spool
/dev/mmcblk0p4            1.0G     23.7M    898.7M   3% /var/log
/dev/mmcblk0p4            1.0G     23.7M    898.7M   3% /etc/ssl/private
/dev/mmcblk0p4            1.0G     23.7M    898.7M   3% /etc/ssh
/dev/mmcblk0p4            1.0G     23.7M    898.7M   3% /etc/nextgen
/dev/mmcblk0p4            1.0G     23.7M    898.7M   3% /home

cat /var/log/messages works and gives a lot of information, e.g. a daemon is running on user "nextgen" called Risco Service Manager.

It was a fun exercise to try to get into a device you own, but don't have remote access too. Next steps could be to try to install extra services on the device that could interact with the Risco Service Manager, or to intercept messages to or from the Risco Service Manager.

Published with permission from Risco

Functional Javascript Update

M.D. van Es — Mon, 09 Mar 2015 23:00:00 GMT

With a background in functional programming (Haskell), I'm interested in bringing functional programming to the web. Frontend Masters offers a course called "Hardcore Functional Programming in JavaScript" by Joe Nelson and Brian Lonsdorf.

For the uninitiated in Functional Programming, you might already be using it without knowing it. If you use jQuery callbacks, e.g. $.get, this uses the functional concept of Higher Order Functions, because it's a function that takes or returns a function. Also the JavaScript native utility Array functions map, filter, reduce etc. are higher order functions and are well known features of functional programming. The course "JavaScript: From Fundamentals to Functional JS" by Bianca Gandolfo can be recommended as an introduction to these concepts and in general to get a deeper insight into JavaScript for intermediate developers. Also see the repo.

A simple example of a Higher Order Function (add), binding it to a new function (add5) and applying the new function:

var add = function(num1) {
  return function(num2) {
    return num1 + num2;
  }
}
var add5 = add(5);
add5(3); // result is 8

"Hardcore Functional Programming in JavaScript" outlines the "point free" style of programming, bringing Functional JavaScript closer to a purely functional language like Haskell. This is also what makes the functional "hardcore" in the case of this course, because the style looks quite foreign to imperative/object-oriented programmers.

Advantages of point free, functional JavaScript include:

It makes functions easier to test. You should need no mocks, because the point free functions are very loosely coupled and highly cohesive (if I understand well).
It's very convenient for working with streams of data/events. This is literally in the description of the Functional Reactive JavaScript lib Bacon.js.
It offers a very different approach to async tasks. In a sense the functional concept of "Futures" provides the same feature as a Promise does. For example, two asynchronous calls that depend on each others result would be written like this (source):

// Future
Future.of(loadPage)
    .ap(Api.get('/products'))
    .ap(Api.get('/reviews'));

// Compare to Promise
request.get({
    url: '/reviews'
}).then(function(response) {
    return request.get({
        url: '/products'
    });
}).then(function(response) {
    loadPage();
});

A disadvantage is that this style of programming in JavaScript is still in it's infancy. The final demo from the course uses 7 libraries, most with some degree of overlapping functionality. We should feel encouraged to help improve the state of these libraries!

The libraries used in the demo:

jQuery
lodash
ramda
pointfree
future, part of this, for as far as I can tell
bacon
socketio

The lack of (static) typing in JavaScript really becomes a large disadvantage when attempting functional programming. I can tell from personal experience that trying to do without typing in Haskell is not worth the headache. The use of a typed transpiler like TypeScript might be of added value, if it's compatible with the Functional JavaScript libraries.

In conclusion, it is possible to do "hardcore" functional programming (i.e. in a Haskell, point free style) in JavaScript, but it's definitely not as easy to pick up as jQuery or even as Angular. You really need to know why you would want to try (I'm not entirely convinced myself), but it would be possible to mix it in with the existing frameworks in your everyday work. On the other hand, it would be advisable to start using the much more accessible functional Array utils, like map, filter, and reduce right away.

It might be worth to mention that the presenters of the course are different from other presenters in my experience. They clearly care strongly about the abstract, academic aspect of programming. During the presentation, they develop in a terminal in Vim instead of in an IDE like Webstorm and don't lint their code, resulting in missing semicolons and otherwise sloppy code. It can be argued that it's not relevant to the subject, but I try to be consistent in my workflow, especially since people will use it as a start point. Also, I enjoy learning about new techniques/tools as a side effect of following these kind of courses.

As a follow-up to this course I will probably look into Lenses (video, nanoscope library), the book "Mostly Adequate Guide to Functional Programming" by Brian Lonsdorf (one of the presenters), the Bacon.js library and the Fantasy Land spec.

Full update for Broek

M.D. van Es — Wed, 11 Feb 2015 23:00:00 GMT

2-12-2015 After updating some packages, I rebooted for the first time after a long time. I thought it would help with cleaning up unused kernels on the boot partition. The box apparently hangs on boot, but no keyboard/monitor are connected so this can still be anything. After connecting some peripherals, nothing happened. Upon trying to reboot the power LED stays on and doesn't respond to pressing (and holding) the power button. Removing and reinserting the power cable is the final straw: now nothing happens, even the power LED stays off.

3-12-2015 In a last attempt to find the culprit, I disassembled the box and plugged in old PSU. Upon connecting power and switching the PSU on, it's fan twitches, but does not start spinning normally. Pressing the power button on the motherboard has no visible effect. The motherboard has probably died.

Building a replacement was already planned, but now has to be rushed unfortunately, so these parts were ordered:

(mobo) ASRock B85M Pro4
(ram) 2x Crucial DDR3 DIMM 1x4GB 1600 C11
(cpu) Intel Celeron G1840
(case) Cooler Master Silencio 352
(fan) Noctua NH-L9i
(psu) Seasonic G-Serie 360Watt

Additionally, I will re-use the drives (80GB + 3TB) and a Samsung 840 SSD (128GB) that I have lying around.

12-12-2015 After assembeling the parts and just popping in the hard drives the system boots normally, although getting the secondary drive to work this requires fixing its entry in /etc/fstab.

The ethernet adapter is not working, which makes sense, because it is a different on-board adapter.

I can't ping outside the system
The ethernet lights next to the port are on
ifconfig lists a br0 and an lo device, but no eth0
lspci lists the Ethernet Controller as Intel I217-V
running sudo lshw -C network gives *-network DISABLED
running dmesg | grep eth gives eth0: link is not ready
renamed network interface eth0 to eth1, then ran:

sudo modprobe e100
sudo ifconfig eth1 up

did not work, but no errors.
According to this article I need to edit /etc/udev/rules.d/70-persistent-net.rules (as root) and comment out the old ethernet device
After rebooting, it detects the ethernet adapter automatically.

13-12-2015 Of course, since the on-board sound card also changed, its drivers also needs to be reconfigured. I plugged the speakers into the lime port, I just want to use 2 channel sound.

aplay /usr/share/sounds/alsa/Front_Center.wav gives an error.
sudo aplay -l does lists several cards, one being PCH ALC892 analog
when using alsamixer, I see that HDMI is set as the default out. The old motherboard did not have HDMI out, so it did detect this new HDMI correctly. This should be changed to analog.
I tried several things that had no effect, such as reinstalling alsa and rebooting, and changing the default device with pacmd list-cards
Eventually, running lspci -knn|grep -iA2 audio shows I'm using sndhdaintel kernel driver
I edit /etc/modprobe.d/alsa-base.conf as root
I add the line options snd_hda_intel index=1 under the last "install" statement
Now I run sudo alsa force-reload
run alsamixer: now analog is active!
run aplay /usr/share/sounds/alsa/Front_Center.wav. I hear the sound!

The case and fans were picked for being very silent. Unfortunately, they are still quite loud at the moment. I did have the sense to pick a case with PWM fans, so I need to configure those.

This article describes how to set it up.

Install: sudo apt-get install lm-sensors fancontrol
Configure lm-sensors: sudo sensors-detect

With the result:

#----cut here----
# Chip drivers
coretemp
nct6775
#----cut here----

After starting the service, sensors gives output. I see coretemp around 30 C and RPMs for fan1, fan2, fan5.
Configure fancontrol: sudo pwmconfig

Found the following devices:
   hwmon0/device is coretemp
   hwmon1 is nct6776

Found the following PWM controls:
  hwmon1/pwm2           current value: 165
  hwmon1/pwm3           current value: 165
respond to changes, pwm1 doesn't. Probably the two chassis fans.

I have a Noctua NH-L9i CPU fan so I could use the Low-Noise adapter, but lets first try PWM.

Fan2 seems to be the CPU fan, because the fan speed corresponds with the max speed of the Noctua NH-L9i. During pwmconfig I found out that setting the speed below 2000 RPM will greatly reduce the sound production. This means the low-noise adapter would definitely help. The CPU fan is hwmon1/fan2_input and is controlled by pwm hwmon1/pwm2.

After configuring both fans with pwmconfig, I start fancontrol service: sudo service fancontrol start and the noise dropped significantly!

The front fan (I think, I can't really hear it that well) is still quite noisy. It doesn't respond to configuring by pwmconfig, for some reason. I might just turn it off. The state after configuring:

fan1 (front?): was 1100 before configuration → after configuration 1100 (unchanged)
fan2 (cpu): 2500 → 1150
fan3 (rear?): 1100 → 380

15-12-2015 Added the low noise adapter to the front fan. Running sensors:

fan1 (front): 1100 → 881.
fan2 (cpu): 980
fan5 (rear): 327.
Cpu temp around 32 C

The noise production is now is very low.

I've also added two extra drives (an SSD and an old 1,5TB HDD) that need to be configured. Running sudo lsblk -o NAME,FSTYPE,SIZE,MOUNTPOINT,LABEL gives:

NAME                    FSTYPE        SIZE MOUNTPOINT  LABEL
sda                                   2.7T            
`-sda1                  ext4          2.7T /mnt/disk3t
sdb                                  74.5G            
|-sdb1                  ext2          243M /boot      
|-sdb2                                  1K            
`-sdb5                  LVM2_member  74.3G            
  |-broek-root (dm-0)   ext4         71.8G /          
  `-broek-swap_1 (dm-1) swap          2.5G [SWAP]     
sdc                                   1.4T            
`-sdc1                  ext3          1.4T             OldDrive
sdd                                 119.2G            
|-sdd1                  ext4          119G            
`-sdd3                  vfat          249M             EFI

The filesystem sizes give away which drives are which and even the label of one of the drives is still intact. For the moment I mount them manually:

sudo mount /dev/sdc1 /mnt/disk1500g
sudo mount /dev/sdd1 /mnt/diskSsdTemp

Both seem to work, so I just added them to the fstab and configured the 1,5TB drive as an export via Webmin.

At this moment only the USB device Rekelbox doesn't work anymore. I thought it might need an update of the USB drivers, not unlike the audio and network drivers before. The initPort.sh script fails with stty: /dev/ttyUSB0: Inappropriate ioctl for device. Trying to find a solution gives many hits, non of which seem appropriate for my case. Trying a USB flash drive works fine, so I don't think it's the USB drivers.

After searching messageboards for a long time, I just tried rebooting with the USB device plugged in: it started working immediately!

Reducing cyclomatic complexity with functional operators

M.D. van Es — Thu, 09 Feb 2017 23:00:00 GMT

Reducing the cyclomatic complexity in a JavaScript application is a good, quantifiable way to make code more readable and maintainable.

Cyclomatic complexity increases as more control statements are used. For a summary of the factors that influence complexity, see "Complexity for JavaScript". Because it is quantifiable, it can be automatically validated in build setups with eslint. Bear in mind that a single switch statement with a large amount of options will have a high cyclomatic complexity but probably is still perfectly readable. As always, rely on common sense when refactoring.

Control statements that concern one parameter often use switch, for example:

const foo = 'banana';
switch (foo) {
    case 'apple':
        console.log('First');
        break;
    case 'banana':
        console.log('Second');
        break;
    case 'coconut':
        console.log('Third');
        break;
    default:
        console.log('Last');
}
// Logs "Second"

A very common way to reduce the cyclomatic complexity with switch is to use a JavaScript object as a map. Instead of traversing the switch statement, do a lookup in the map where the key is the condition and the value is the action. For the example above this could look like this:

const actions = {
    apple: _ => console.log('First'),
    banana: _ => console.log('Second'),
    coconut: _ => console.log('Third')
}
const foo = 'banana';
if(actions.hasOwnProperty(foo)) {
    actions[foo]();
} else {
    console.log('Last');
}
// Logs "Second"

This does not work directly with more complex conditions. e.g. a composite condition that checks against multiple parameters within one statement or between statements.

function example(error, stdout, stderr, mapping) {
    if(stderr && stderr.length > 0 && stderr.indexOf('A problem occurred') > -1) {
        console.log('Not found');
    } else if(error || (stderr && stderr.length > 0)) {
        console.log(`Execution error ${[error, stdout, stderr].join('|')}`);
    } else if(!Array.isArray(mapping)) {
        console.log(`Invalid configuration ${[error, stdout, stderr, mapping].join('|')}`);
    } else if(!stdout || stdout.length <= 0) {
        console.log('No output');
    } else {
        console.log('Result from some process with mapping and stdout');
    }
}
example(null,null,'A problem occurred'); // Logs "Not found"
example(new Error()); // Logs "Execution error Error||"
example(null,null,null,null); // Logs "Invalid configuration |||"
example(null,'',null,[]); // Logs "No output"
example(null,'Foo',null,[]); // Logs "Result from some process with mapping and stdout"

In this case a JavaScript Array of objects can be used here, with one property holding the condition and another holding the desired action.

// First part
  const actions = [{
    condition: _ => !stdout || stdout.length <= 0,
    action: _ => console.log('No result')
  }, {
    condition: _ => !Array.isArray(mapping),
    action: _ => console.log('Config invalid')
  }, {
    condition: _ => error || (stderr && stderr.length > 0),
    action: _ => console.log('Error executing')
  }, {
    condition: _ => stderr && stderr.length > 0 && stderr.indexOf('A problem occurred') > -1,
    action: _ => console.log('Not found')
  }]

It would be possible to loop over each entry with a for loop or an iterator (see an example), but this is where the expressiveness of the functional operators Array.prototype.filter and Array.prototype.reduce shines.

// Second part
function example(error, stdout, stderr, mapping) {
  const actions = ... // See above

  const selectedAction = actions
    .filter(action => action.condition())
    .reduce((accumulated, currentAction) => {
      // By design, reduce will only keep the last match when the accumulator 
      // is ignored, so the order inside the actions array has significance.
      return currentAction.action;
    }, function() {
      // Else clause
      console.log('Result from some process with mapping and stdout');
    });
  selectedAction();
}

example(null, null, 'A problem occurred'); // Logs "Not found"
example(new Error()); // Logs "Error executing"
example(null, null, null, null); // Logs "Config invalid"
example(null, '', null, []); // Logs "No result"
example(null, 'Foo', null, []); // Logs "Result from some process with mapping and stdout"

A working example can be found in this fiddle.

This shows how to use functional operators to reduce cyclomatic complexity for an if else condition. Do keep in mind it may not be beneficial for performance. For instance, do not declare the action array inside of the function as in the example, as it would redeclare the array each time the function is executed. Another improvement would be to stop iteration in filter/reduce as soon as a matching action was found.

Also note that the example with the if else condition itself is quite readable, so it will not be worth the trade off of refactoring to resolution through an array of objects in this case.

Apollo Client in Practice

M.D. van Es — Mon, 04 Feb 2019 23:00:00 GMT

Some time ago I joined a team that is working on a search application. The application takes search terms and displays the results in a table with potentially dozens of columns and hundreds of rows, even before pagination. It is implemented in React and uses Apollo for GraphQL calls. I was surprised to find it noticeably slow when a lot of search results were retrieved. React is well-known for leveraging virtual DOM to optimize performance and GraphQL should even be able to add caching to further optimize performance on the side of network requests.

Looking into the performance tab of Chrome dev tools lead me to believe the performance problems were caused by computations in the bottom components (e.g. formatting in cells). Because there are so many and they are re-rendered quite often, this approach is quite intensive on resources.

Besides that, the application had obvious state synchronization problems. When moving between views it was not maintaining the same state of selected rows. Even though Redux was used to store application state and communicate it between components, it was not used consistently. There were still plenty of React class components that stored some parts of the state locally.

To summarize, there were two issues that needed to be solved:

Poor performance due to excessive re-rendering
Loss of application state when navigating views due to decentralized state stores

Since both issues were caused by (a lack of) architecture, we redesigned the structure of the application. The original implementation used:

Apollo Client as a GraphQL client
Axios as an HTTP client for REST endpoints
Redux and React local state to manage the state between components

It used Apollo, but by manually firing client.query() and after processing the response, it stored the result in the Redux store.

Fixing application state with Apollo Local State

When restructuring the application, Apollo Client was updated to 2.5. This version has a built-in local state manager (formerly apollo-link-state) and it supports REST calls with the apollo-link-rest plugin. The apollo-boost package contains the client and several useful plugins. Adopting these means that both Redux and Axios can be removed and Apollo will be used as a single source of truth. If there is a single store for the data, there is no need for synchronization and with that one of the issues is solved.

The way we used Apollo Client was also updated, to create a better separation of UI and data. Instead of using client.query() directly in the component lifecycle methods, components are split into a presentational component and enhanced with the graphql() HOC to add data from remote (i.e. GraphQL back-end) or local fields. Both utilize the Apollo cache, which fulfills multiple functions, one of them an application local state store.

Example of wrapping a component in a Query HOC:

const Books = ({ data: { books } }) => (
  <ul>
    {books.map(book => <li>{book.title}</li>}
  </ul>);

export graphql({ query: gql`
  query($author: String!) {
    books(author: $author) {
     title
    }
  }`, 
  variables: { author: "Mickiewicz" })(Books);

Apollo reactively updates when using Query as a container, basically like the connect HOC in Redux. When the variables prop on the Query component is updated, it will automatically re-query. It uses the cache if possible and falls back to a network call if needed, although this behavior can be configured.

Improving performance with Local Field Resolvers

Having a single source of truth fixes the state synchronization problem. It also paves the way for improving the performance. In general, when a lot of data enters the application, it is a good idea to format it once and cascade the formatted data down to the components and it's descendants with as little transformations to the data itself. This reduces the amount of computations in the lower components, which solves our other issue.

When using Redux, a common way to transform data in the store is using Reselect, which computes derived data from the Redux store with selectors. For Apollo this is done by:

wrapping the table in a Query that queries a local prop rows @client, using the @client directive
making client side resolvers for rows that queries GraphQL endpoint
mapping the data in a resolver from a raw format to a format ready for the table components, e.g.:

from a data object

{ 
  author: "Mickiewicz", 
  publications: [ 
    { 
      title: "Pan Tadeusz",
      date: -4291747200 
    } 
  ] 
}

to an array rows of cells

[
  [ "Mickiewicz", "Pan Tadeusz", "January 1834" ]
]

Next steps

Apollo is excellent for merging data from multiple sources (in this case GraphQL, REST, local state and cache) and functions as a "single source of truth" which should solve the state synchronization problems. The local fields that Apollo uses in its local state manager can derive data, moving expensive operations from component render functions to resolvers in its application level cache. Although the issues mentioned in the introduction are now dealt with, we did encounter plenty of other issues I may dive into later. However, these are some things that you might want to take into account when working with Apollo Client:

Outside of restructuring the application, we improved performance with react-virtualized which speeds up rendering large tables. Apollo also offers GraphQL pagination. We did not use that, as we have to do our pagination on the client side to keep the sorting feature of react-virtualized in tact.

Apollo Client offers support for TypeScript, it is even possible to generate queries and typed React components from GraphQL schemas with @graphql-codegen/cli.

Also definitely use the JS GraphQL IntelliJ Plugin because it will not only auto complete queries, but it will help you think about (client side) schema's.

When the Query component mounts, it creates an observable that subscribes to the query in the query prop. This encourages reactive behavior like RxJS (which can also be used as a state store). However, it seems that Apollo offers much less fine-grained control over the observables than what RxJS provides. And considering observables, Apollo Client effortlessly scales to web sockets!

Are you looking for inspiration on how Apollo client can be applied? I can recommend this talk by Uri Goldshtein and this introduction to Apollo state management by Sara Vieira.

Dop won't boot and USB issues

M.D. van Es — Mon, 02 Feb 2015 23:00:00 GMT

When I plugged in an USB stick in the USB3 front panel of my workstation dop, there was no response. The USB stick's light didn't turn on.

After I opened up the sidepanel the USB plug to the motherboard wasn't plugged in, so I fixed that.

After rebooting, I was greeted with this ugly message:

Error: no such partition found 
grub rescue>

I tried using grub rescue. But using ls I can't find the boot partion. Then I notice, the bootdisk isn't showing up in the EFI.

After opening up the sidepanel again I notice the power is not plugged in to my SSD, the bootdisk of the machine... Strangely, after plugging it in, the same error occurs.

Booting from an Ubuntu live CD, all drives are visible, but the main SSD (/dev/sda2) shows an error in gparted. It does have the bios_grub flag. It gives the error:

unable to detect file system

After deleting the /dev/sda4 bios_grub partition of 17MB this leaves an EFI fat32 partition with boot flag (dev/sda3) and main partition /dev/sda1 with /boot.

Reboot: doesn't change anything.

Now I'm really need to use the system, so I try rebooting with super grub disk. The system boots immediately with first found settings. I think I'll leave it like this because I want to upgrade the SSD in the near future anyway.

Utopic Dop

M.D. van Es — Thu, 01 Jan 2015 23:00:00 GMT

Upgrade started

Replaced /etc/gnome/defaults.list
During installation the "report problem" pop-up appears, but the upgrade continues. I press report problem and it's about i915-3.15-3.13-dkms. Seems to be Intel graphics related.
Changes to php.ini, kept my version
PhpMyAdmin is updated. Had to give root db password which was difficult to find... Noted it now.
After reboot the system seems to run fine.

Third party repos that were disabled on upgrade to utopic:

Installed new repo:

http://download.opensuse.org/repositories/isv:/ownCloud:/desktop/xUbuntu_14.04/

Re-added:

Not re-added:

Update 7-2-2015

When trying to update owncloud, this message appears:

The following packages have unmet dependencies:
 owncloud-client : Depends: libowncloudsync0 (= 1.7.1) but it is not going to be installed
                   Depends: owncloud-client-l10n but it is not going to be installed
E: Unable to correct problems, you have held broken packages.

Install from .deb should work, but it complains about a conflict with the installed package 'libqtkeychain0'.

Running:

sudo apt-get update
apt-get autoremove

removed owncloud repos from Other Software
removed libqtkeychain0 0.20140128 with Ubuntu Software Center
now libqtkeychain0 0.3.0-2 is available -> installed it
re-added owncloud repo and run installation -> works!

Now there still is a problem with lxc-docker, it can't be updated.

apt-get remove lxc-docker
apt-get install lxc-docker

This works!

MDWorld

Polymer 2 and Redux

Dop SSD update

Making bootable

Moving /home

Micro Frontends In A Nutshell

Micro Frontends In A Nutshell

Why use Micro Frontends?

What are Micro Frontends?

How do you build Micro Frontends?

When to use Micro Frontends?

Want to know more?

Recommended in-depth reading

On Reactive

Reactive Programming

Functional Reactive Programming

React

Ractive.js

Reactive Extensions (a.k.a. RxJS)

Flapjax

Functional JavaScript

LoDash

Real-time web

Responsive/Adaptive design

NBI Quickscan app

Elm for front-end developers

Why Elm?

Immutability

Toolchain/workflow

Elm-reactor and external CSS

Webpack

Getting the debugger back

Unit tests

Speeding up the build on Travis

Combining several components in one Elm app

Conclusion

Fronteers 2014: CSS

Icon Font

Basic CSS

Visual Representation of the CSS feature box-sizing

Visual Representation of the CSS features float and clear

Discourse on Ubuntu 14.04

Fronteers 2014: A Glossary

A Spacecraft for All

BlobAPI

CSS Animate

Dialog

ES6 Compatibility Table

Flexy Boxes

getUserMedia

Intern

Istanbul

JSComplexity

Line-mode browser

Penthouse

Phaser

Plato

ServiceWorker

Shadow DOM

Spritesmith

SUIT CSS

SVGInjector

Traceur

Ultimate CSS Gradient Generator

Webpack Module Bundler

WebPageTest

WebRTC

will-change

Xibalba

Yeast

Dop Beeping HDD

npm workaround

Node on RPi

Compiling Node on RPi

Site updated

Cover images

Old themes

Polymer 2 and TypeScript

Webpack

Adding ts-loader