BBQ with a Dutch ICT/start up Delegation

So a few weeks back I received an interesting email that at first thought looked like a scam.

In short I was being invited to a BBQ at the Consul General’s residence in Sydney to meet an ICT/start up delegation from the Netherlands. Let me straight up say I have no idea why I would receive something like this. I am not dutch, have no dutch ancestry (that I am aware of) and while somehow is listed on the top startups in Sydney somehow its not as though it has lots of press coverage or is a real start-up (its more a hobby side business at the moment).

You can view the email above. I think its the use of _ in the title that triggered my personal spam filter. However it was not flagged as such by Gmail, the headers all looked fine and when I followed through the various links it all seemed above board. I posted it in the company slack channel asking what others thought. There was a bit of debate, but I figured I would follow it up on twitter with the person who sent to email.

They actually responded back that yes it was real and if I could respond to the email. Which I did.

A short while later and the details were sent though. Monday 16th October 2017 at an address in Seaforth (swanky!). There was some discussion about if this was indeed a real event and that I should have an escape plan.

Assuming it was real I had two goals. The first was to find out why I was invited. The second was to find out why they were interested in Australia.

The day came around and I took a taxi driven by a racist who hates uber (the guy driving and his stories are worth his own blog post) and I arrived.

I had no expectations going in. I have never attended anything like this in my life. My plan was to just roll with the punches and see what I could learn and perhaps meet some interesting people.

Have you ever had that moment where you realised not only are you the most poorly dressed but you also just dragged down the average age by being an outlier? It was one of those meetings. I quickly noticed that I in jeans and hoodie was really underdressed by most who were wearing suits. I guess I could have played it off as though I was the next Zuckerberg though. I was welcomed by Willem (the Consol) a very nice chap who greeted me in what I presume was Dutch but frankly could have been English as I was so bewildered. He swiftly invited me in and offered me a beer. At the time I was more thirsty and requested water, and pointed out tap water was fine. Probably a mistake on my part. Here I was in a fancy house/mansion and asking for a glass of tap water.

Anyway after some brief chatting I noticed Nicoline who was the one who invited me. Not familiar with Dutch culture and frankly not being good at reading people it was hard to see what she thought of me. I suspect surprise at my appearance and age perhaps? I can only apologise for not being more impressive. I didn’t get the chance to say hello at this point as I was ushered outside.

The view of the house was excellent and while the backyard was small due to the presence of a pool in the middle of it it was quite pleasant. After walking outside I noticed that Paul Budde just happened to meander outside. I did a double take for a while before realising it was indeed him, especially when he pronounced his last name. Spotting some individuals more my own age I went over and introduced myself. Turns out they were interns working at the consulate and dragged into cooking duties. I ended up chatting to them for a while learning that Holland exports quite a lot of natural gas but is also starting to stop this process due to it causing the land to sink further.

Around this time Willem gave a quick speech. I got the impression that the Dutch look at Australia as a missed opportunity, considering that they found it 100+ years before the English but never did anything with it. They were quick to point out that it was called New Holland for a time. Highlight of the speech was the arrival of a Kookaburra who was eyeing off the BBQ goods and the surprise of the visitors to Australia at the native fauna.

Following the speech there was a “round table” introduction of who you were and why you where there.

Awkward. Thats the word that best describes me at this point. After Paul Budde, everyone else was Directors of companies, CEO’s, Founders, Co-Founders etc… At my time to say who I was I decided to go with the technique I believe is called a purple elephant. I mentioned who I work for (Kablamo) what I do (programmer working with AWS) and that I run on the side as well. I finished by saying “And I don’t like to eat raw tomato”. No idea why I use that but it tends to be easy to remember and makes you stand out. No longer was I the uncomfortable poorly dressed guy, I was the uncomfortable poorly dressed guy who doesn’t like to eat raw tomato. It did prove a talking point when the salads were being served with a few individuals pointing out I should steer clear of the middle one due to the presence of chopped tomato.

Following the round table there was a call for a group photo. As such there is some photo of an ICT startup delegation in the backyard of the Dutch consul generals house with a scruffy Australian in the back. I made sure to place myself in it such that it would be easy to photoshop me out. I’d love to get a copy of the photo though.

I figured at this point I would attempt to mingle, went to grab a beer and with that in hand start chatting away. Paul Budde of who I have read a lot of his work regarding one of my pet hates in Australia the mismanagement of the NBN. I decided there and then to have a conversation with him. After a swig of liquid courage I walked up offered to shake his hand and mentioned I was a big fan of his work, and especially interested in his comments about the NBN. I managed to hold a 15 minute conversation with him about the NBN without feeling like an intellectual numpty. Interesting takeaways from the conversation were that he thinks it will be 2030 before anything is done to fix the current situation, and that he has talked to the Prime Minister Malcom Turnbull on it and he believes that Turnbull 100% knows he is doing the wrong thing but has to continue for political reasons. I managed to suggest a beer to him from the selection available which he agreed was a good choice after tasting it.

Following this the party moved inside around the kitchen. I ended up at this point chatting to a few people the main one being Ruud Alaerds the director of Dutch Hosting Provider Association DHPA. He asked me about how hosting companies are in Australia and how they are going with being acquired. He was interested in know if the local companies are merging into a super-company and pointing out that companies such as OVH are probably waiting for most of the local companies to gobble each other up and then acquire the winner. I believe he is looking to get in before this. He was also able to answer my second question of why Australia? Turns out Australia is seen as a good stepping stone into Asia as Singapore is already “full” and the other places such as Taiwan and Hong Kong are not worth investigating at this point.

Other conversations I had were interesting. Things like discussions about houses in Double Bay (very swanky) lead to some asking where I live (near Berowra) and after explaining where it is there were comments such as “Oh that must be nice for you”. Which in fairness by comparison it is a very humble place to live.

Eventually the desert was offered (small cheesecakes) a small thank you speech and people started leaving. At this point I went for the direct approach. I walked up to Nicoline and thanked her for the invitation and again asked why was I invited. Her response was not really encouraging, just that she was looking around for local people involved in the start-up community in Australia. I guess that since I have worked for start-ups and have what is considered one that might be why I popped into her list, but I suspect ultimately it was more a mistake. I did have an enjoyable evening regardless!

All in all Boyter’s uncomfortable night out with the Dutch ICT/start up Delegation was quite entertaining. While I didn’t get any actionable items out of it I did enjoy meeting Paul in particular and seeing how people who are considerably more successful that I am live.

8/10 would do again. Much thanks to the very kind individual who gave me a lift out of there as well which got me home before 11pm. I still believe I was invited by mistake and that while there I was more of a dancing monkey than someone who was able to contribute but none the less it was very interesting for myself.

Home Battery Systems – You may de-rate system capacity

A quick post. I was looking at the prices to install a home battery system the other day. During the sales process I was informed that for a modular system such as the one provided by Ampetus that if you don’t buy 2 or more batteries that

“you may de rate system capacity”

Sound’s impressively scary! Slightly worried I asked that it means. Thankfully what it actually means is that instead of getting 3kW delivery using two batteries that one battery matched with the inverter may only deliver 1.5kW. This is not a problem if you are still connected to the grid but it can mean that your inverter isn’t working at its full potential. So in short nothing to worry about.

Why is this GoLang solution faster than the equivalent Java Solution?

The below is a small story about how I took a program with a runtime of hours to minutes, to tens of seconds to seconds. More for my own personal amusement then anything else.

At work there is a tradition of a Friday quiz being posted by the winner of the previous week. I missed out on the most recent one due to having to duck off early to do my tax but the problem was rather an interesting one.

The challange itself is not as simple as you would initally think and taken from a 2015 IBM Ponder This

Three people are playing the following betting game.
Every five minutes, a turn takes place in which a random player rests and the other two bet
against one another with all of their money.
The player with the smaller amount of money always wins,
doubling his money by taking it from the loser.
For example, if the initial amounts of money are 1, 4, and 6,
then the result of the first turn can be either
2,3,6 (1 wins against 4);
1,8,2 (4 wins against 6); or
2,4,5 (1 wins against 6).
If two players with the same amount of money play against one another,
the game immediately ends for all three players.
Find initial amounts of money for the three players, where none of the three has more than 255,
and in such a way that the game cannot end in less than one hour. (So at least 12 turns)
In the example above (1,4,6), there is no way to end the game in less than 15 minutes.
All numbers must be positive integers.

Only one person managed to find an answer, lets call him Josh (because that’s his name), having spent a few hours writing up a solution using his favourite programming language Go. Come Monday morning I arrived, looked at the quiz I and became intrigued. Could I write a version that would outperform his. After all Go is a pretty performant language, but I suspected that he may have missed some easy optimisations, and if I picked something equally as fast I should be able to at least equal it.

I took a copy of his code (since modified to be much faster) with a runtime of 1 minute 40 seconds (on my laptop) started work.

Looking at the problem itself we have a few pieces of information we can use. One is that we don’t ever need to calculate the 12th turn. If the game makes the 11th turn and still continues then it made 12 so that saves us some calculations. Another is that if any money amounts are the same we can stop the game instantlty, and more importantly not even add those to the loop.

Given that the existing solution was written in Go it seemed insane to some that I started writing my solution at least initally in Python. This however is not as crazy as it seems. Python being higly malleable allow some rapid iteration, trying out a few things before moving over to another language.

The first thing to note is that you need to generate a tree of every possible combination that a game can take. Then you iterate over each one for the inital starting amounts of money to determine if the game ever ends, and if not mark it as a game that does not finish. To generate the combinations I went with a fairly simple recursive strategy,

# Calculate all the possible sequences for who misses out for each turn
def calc_events(current=[], turn=0):
    if turn == DESIRED_TURNS:
        return [current]

    one = list(current)

    two = list(current)

    three = list(current)

    turn += 1
    path1 = calc_events(current=one, turn=turn)
    path2 = calc_events(current=two, turn=turn)
    path3 = calc_events(current=three, turn=turn)

    return path1 + path2 + path3

The result of running the above is an array of arrays containing each situation where someone sits out a turn. It is important to note that his produces a list that modifies from the back (big-endian so to speak) as this will be a very important consideration later.

The result looks something like this (truncated to just 4 results),

[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1]
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2]
[0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0]

Given the above I coded up a simple solution which simply looped through the combinations. Here it is in pseudocode.

for player1 money:
  for player2 money:
    for player3 money:
      for game in games:
        for turn in game:
          result = play turn using money amounts
        if result:
          print player1, player2, player3

The above is about as bad as it gets for an algorithm as we have 5 nested loops. As expected the runtime performance was horrible. In fact I added a progress bar which simply calculated how far into the first loop the application had reached. Using Python I worked out that it was going to take several hours to get a result using the above. Not good enough.

A few calculations and I realised that I was asking my poor program to calculate something like 50 billion games. Clearly I needed to reduce the number of games and speed up the processing as much as possible. The first change I made was to only calculate the game turn events once and keep the result. I had also written the code to be as readable as possible, which how you should write anything but as this can be an issue also removed the following function by inlining it for improved performance.

def turn(p1, p2):
    if p1 == p2:
        return False, p1, p2

    if p1 > p2:
        p1 = p1 - p2
        p2 = p2 + p2
        p2 = p2 - p1
        p1 = p1 + p1
    return True, p1, p2

The next thing I did was generate all the permutations of money amounts to play games with into a single list. The last thing I did was switch from using Python to PyPy which with its JIT should speed up the loops considerably. The result of all this can be found here and its runtime using PyPy dropped to ~8 minutes.

8 minutes was about 5 times slower then the GoLang program at this point, which is pretty good for a dynamic language like Python, and consider that my implementation was single threaded where as the Go was using as many cores as it could get. My next step was to implement the same program in a faster language. The only other faster languages I know that I have any experience in are C# and Java. Since I already had Java setup I went with that. I will also admint it was about proving a point. Go may be fast, but at writing Java should be able to equal it for most tasks.

At this point however I mentioned to Josh that his Go program had some inefficiencies. The big one being that he was calculating the 12th game needlessly. I then modified his program and with some other changed reduced the Go program runtime down to ~40 seconds. I was starting to get worried at this point, as I was aiming to beat the Go program’s performance by 5%. No idea why I picked this number but it seemed reasonable if I was smart with the implementation.

At first I ported the Python program in its original readable reusable form to Java and ran it. The runtime was ~7 minutes. I then inlined the same functions and converted it over to use parallel streams. This time the runtime was about 90 seconds. This would have been fast enough had I not mentioned to Josh how he could improve his code. I had shifted the goalposts on myself and had a new target now of ~40 seconds.

After doing a few things such as inlining where possible, changing to enum and some other small tweaks I had a runtime of ~60 seconds. The big breakthrough I had was after looking at the hot function I realised that storing the game events in a List of Arrays meant that we had a loop in a loop. It was however possible to flatten this into a single array of integers and reset the game every 11 turns with a simple if check.

This was the breakthrough I needed. Suddently the runtime dropped from ~60 seconds to about 23 seconds. I happily posted my success in the Slack channel with a link to the code and sat back.

The smile was soon turned to a frown. Josh implemented the same logic into his Go program and it now ran in ~6 seconds. It was suddently 5x times faster. At this point I had a mild panic and started playing with bitwise operations and other micro optimisations before realising that no matter what I did he could simply implement the same change and get the same performance benefit as we were both using roughtly the same algorithm.

Something was clearly wrong. Either Go was suddently 5x faster at a basic for loop and integer math then Java OR I had a bug in my code somewhere which was making it worse. I looked. Josh looked. A few other people looked. Nobody could work out what the issue was. At this point I didn’t care about the runtime, I just wanted to know WHY did it appear to be running slower. I was so desperate for an answer I did what all programmers do at this point and outsourced to the collective brain known as Stack Overflow

A few un-constructive comments came back such as Java is slow (seriously its not 1995 anymore guys, Java is fast) etc… Thankfully one brilliant person managed to point out what I had missed. It was not the loop itself, but the input. Remember how I said that the generation of the events being big-endian was important? Turns out the Go program had done the reverse and implemented it little-endian.

The thing about the core loop is that it has a bail-out condition. If two players have the same money amount we end the game and don’t process any further. The worst possible situation for the loop is to process almost every condition only to find out just at the end that it ended. Its a lot of processing work. Ideally you want to find the failing conditions as soon as possible. By changing the games from the end I was forcing the Java program to process about 5x times as many combinations as the Go program.

It just happended to be that Josh has picked a more optimal path through the games.

A simple reverse of the games (line 126 of the linked solution) and suddenly the Java program was running in about ~6 seconds and the same time as the Go program. You can view the code where

For fun I tried running it on a 16 core VPS and it ran in about ~2 seconds and maxed out all the cores so it seems parallel streams do what you expect them to.

Interestingly while Josh’s starting positions was more optimal then mine, its probably still not the optimal path for this problem. There is bound to be a way to generate the game such that you hit the failing conditions as soon as possible saving needless processing. There is probably a Thesis for a PhD in there somewhere.

I figure this is probably as far as I want to take this. I did play around with bitwise operations and loop un-rolling but the time didn’t change that much.

I certainly had fun with the implementation and working things out. Some may argue that optimising for a micro benchmark such as this is a waste of time, and generally they would be right. That said there are occasions where you really do need to optimise the hell out of something, say a diff algorithm or some such. In any case what developer does not dream of saving the day with some hand unrolled loop optimisation that saves the company millions and brings the developer the praise and respect of their peers!

EDIT – A rather smart person by the name of dietrichepp pointed out that there is a better algorithm. Rather than brute force the states work backwards. You can read their comment on Hacker News and view their code in C++ on Github.

The Worst Individual I Ever Worked With

Taken from a comment I posted on HN in a thread about a Soccer Con Man.

Not actually a programmer. The guy was hired to be a project manager.

After joining things were as expected but after a few weeks we noticed that he was rarely around after lunch and never around after lunch on a Friday.

We would email him at those times deliberately to catch him out and I recall starting to put sticky notes on his laptop “Came to see you a X time”. He would come back and just dump all the notes in the rubbish and claim he never got them. He would often claim to be working from home, despite his laptop being on his desk and usually closed. He would also never responding during those times to email or IM.

A classic seagull manager he would appear when something went wrong, making a lot of noise, writing a lot of emails and then vanishing. He would also be sure to be seen when something was delivered often staying back late on those times.

It got so bad one friend of mine started tracking when he was around and then tracking when one of his relatives died. During his tenure the following incidents occurred,

– Hot water system blew up. 4 times. He had pictures which he would show all the time.

– Uncles, aunts, and various over family members died to the total of 20 individuals.

– Our time tracking him showed him to be in the office less than 15 hours a week on average.

We started to suspect he had a second job and was pulling the same con on them. This was never proved, but we did find someone who had worked with him previously and they reported the same behavior.

The worst thing was it was raised with management at least several dozen times and nothing ever happened. He managed to pull this scam off for 4 years. I could not believe the waste of money this guy was, literally $500,000 burnt on a useless individual.

C# as a Language from old Google+ Post

The more I use C# as a language for writing things the more I am convinced that its approach really is the best language approach out there.

The unit test support is excellent which allows development speed to be just as fast as any dynamic language (Python, PHP, Perl).

The static typing catches so many issues before you get to runtime and allows sweeping changes without breaking things.

Unlike Java it has the var keyword (saves time and improves readability) and so many more useful functions which yes you can replicate but are just built in and work correctly.

Then you get to the really good stuff. LINQ is awesome. The lazy loading allows you to implement a repository pattern over your database which is just awesome. Set up the basic select * from then add extension methods allowing you to chain whatever you need, EG

from person in _dbContext.GetPerson().ByUserName(username).ByPassword(password);

100% elegant, easy to test, easy to write, easy to read and understand and generally works exactly as you would expect without any hidden gotchas. And because its lazy it doesn’t chew resources sucking back everything from the database.

You can use functional programming techniques if you wish, and with the new async decorators you can work in a node.js style if you with, with static typing and all existing library support.

Or you can continue to work in a C like manner, or mix it up with objects, procedural code and functional.

I switched back to Java not that long ago to write a simple server using Jetty and even with things like Guice (best DI implementation I have used so far) and Guava it was still painful. Less painful, but I really felt that the compiler was fighting me from doing things in an elegant manner most of the time. Even adding the “var” keyword would improve Java in a massive way. Add some functional programming in there and I would be pretty happy.

I just wish C# would run on the JVM as I would use it for pretty much everything in a heartbeat. As it is the Mono support is missing the stuff I really want and isn’t as seamless as the experience should be. A pity really as C# really is in my experience the nicest language to work today that’s production ready.

Interesting Code Comment

Found the following comment in some code I had modified a few years ago.

Just to set this up, its an existing application I had no hand in creating, and is a totally atrocity of 180,000 lines of untested code (and pretty much un-testable) which through the abuse of extension methods lives in a single class spread out across multiple files.

This is evil but necessary. For some reason people have put validation rules here rather then in the bloody ValidationHelper. Thanks to their incompetence or genius... we now have no idea if we add the extra validation in the correct place and call it here if it will work. Since this is also 180,000 lines of non tested nor testable code (without refactoring) I have no confidence in making any changes. Sure we have subversion but that dosnt allow us to code fearlessly ripping apart methods and refactoring since we have no test safety net.

I guess the obligatory car analogy would be driving down the highway, carrying nuclear waste, in an open container, in a snow storm, with acid/lsd/ice fueled drugie ninja bikies attacking you, while on fire, while juggling chainsaws, and all of a sudden you need to change the tyre. So much is going on that its you dont want to risk it and then when forced to do so 
you know its going to end up badly.

If you are still reading this then for the love of all things holy, help by refacting stuff so we can test it properly. The DAO layer should be fairly simple but everthing else is a shambles. 

Rant time over. Lets commit sin by adding more validation.

Not so unique GUID

I have been doing a lot of work with the Sitecore CMS recently. Once of the things you quickly learn is how it relies on GUID’s for pretty much everything. This means of course when you start testing and need to supply GUID’s into your tests that you end up with lots of GUIDs that look like the following sprinkled through your code {11111111-1111-1111-1111-111111111111}

Today I remarked that we should be using things like “deadbeef” for the first part of the GUID with a colleague. He suggested that we should try and actually write something. With a little bit of 1337 speak this is actually possible. Naturally we got back to work, but with a little free time I quickly coded up a simple Python application to generate “phrased” GUID’s. Some examples follow,

silicles-oafs-blob-tael-declassified -> {5111c1e5-0af5-b10b-7ae1-dec1a551f1ed}
deedless-gait-soft-goes-eisteddfodic -> {deed1e55-9a17-50f7-90e5-e157eddf0d1c}
libelist-diel-alls-flit-disaffiliate -> {11be1157-d1e1-a115-f117-d15aff111a7e}
offstage-diel-labs-scat-classifiable -> {0ff57a9e-d1e1-1ab5-5ca7-c1a551f1ab1e}

None of the above are make much sense, but by looking at the outputs you can attempt to write something such as,

 cassette soft gold dice collectibles

Very zen. Some rough back of napkin calculations gives my program something like 10,000,000,000,000 combinations of GUID’s based on the word list I supplied. I may just turn it into a online GUID generator like this one

EDIT – You can now get these guids at