Tdd Apps

Book Review: Cracking the Coding Interview

Sat, 11 Jul 2020 00:00:00 +0000

Interviews are stressful, they cover rarely used topics. Industry leading companies expect candidates to prepare in many areas. It is impossible to answer every question correctly. Reading a book won’t get you hired but this book is as good of a practice guide as it gets.

Exercises

Cracking the Coding Interview is a good refresher of many Computer Science topics. The book has hundreds of exercises of data structures and algorithms. Every solution is explained in detail.

Big-O notation

The book does a superb job explaining time and space complexity. This topic comes up in every interview. Candidates need to understand it well.

Be prepared

Reading a book will not get you hired. Practice is very important. Do the exercises, practice coding on a whiteboard. Be aware different companies value different skills and you may even get rejected after intensive practice.

Lessons from building a Raspberry Pi 4 NAS

Tue, 05 May 2020 00:00:00 +0000

I built a home NAS with a Raspberry Pi 4. It took longer than expected. The end result is not as great as I originally envisioned. These are some bits of hard-earned knowledge to save you time and pain.

USB

USB 3 support is not great. I tested at least four different drives and as many drive enclosures. Powered USB hubs. Even powered hard drive enclosures. Drives just disconnect regularly.

This forum post helped mitigate some USB problems. However, I found no reliable way to copy hundreds of gigabytes over USB. I had to copy the bulk of the data on my laptop.

File System

Since I had to plug the NAS disk to my laptop, file system selection was harder.

I chose Ex-FAT. It comes native on Mac OS and can be installed on Linux. However, mount.exfat CPU usage becomes high at times.

Operating System

Use Raspbian to minimize friction. Most documentation and examples are based on it.

I tried Ubuntu first. But some things work differently (e.g. /boot/cmdline.txt is in /boot/firmware/nobtcmd.txt).

Power Supply

Get the Official Power Supply. The Raspberry Pi 4 comes with a USB-C charging port. But this port is not entirely compliant with the USB-C standard. Incompatible cables and power supplies cause throttling.

Monitor your computer physical parameters. Voltage, temperature, etc. If the board gets too hot it will get throttled. The vcgencmd is very useful to troubleshoot these issues.

Automation

Automate your setup. I created a few ansible playbooks for provisioning. These playbooks are the NAS core.

I tried many different approaches because I had the automation to reduce variance and manual errors. Ultimately, this NAS will be moved to a different computer and the ansible playbooks will help me do it.

Learn

Building a NAS out of a Raspberry Pi is a good learning experience for those who want to learn more about Linux administration.

Is it worth the time and effort compared to an off-the-shelf solution? It depends.

This NAS is not suitable for high volume loads. But it is silent, small, and has a close to zero power consumption. It is built on a general purpose computer and operating system. It has no proprietary restrictions. It gets regular security updates. It gives you control to tailor it to your needs.

South Florida's Codecamp 2020

Sat, 29 Feb 2020 00:00:00 +0000

These are the notes on my Improve your AWS security presentation

Slides

Source Code

Presentation Outline

Cloud adoption has rapidly increased with the years. The cloud empowers companies to change faster. However, speed comes with its own set of risks: accidental data leaks, internal services publicly accessible, compromised account credentials. AWS offers several services to help secure and monitor your account.

Wouldn’t it be great to automatically validate S3 buckets are not publicly available? Would you like to know who destroyed the test EC2 instance without your consent? Would you like to get an alert when your account is under attack?

This presentation will cover how to use three powerful AWS services to answer these and other questions. We will walk through AWS Config, AWS Cloudtrail and AWS Guarduty. What do they do. And what problems they solve.

This is an introductory talk. Participants with any level of AWS knowledge are welcomed. This is an AWS only talk. The comparable Azure services will be explained in a future presentation.

Solarize Slack for Real

Tue, 19 Mar 2019 00:00:00 +0000

These are the instructions I followed to make Slack look like a Solarized app.

Note: The recent Slack client rewrite renders these instructions unusable.

Before

After

Use the following sidebar theme. Detailed instructions.

#073642,#002B36,#B58900,#FDF6E3,#CB4B16,#FDF6E3,#2AA198,#DC322F

Theme Source.

Step 2: Change the Background Colors

Add the following block to /Applications/Slack.app/Contents/Resources/app.asar.unpacked/src/static/ssb-interop.js

document.addEventListener('DOMContentLoaded', function() {
 $.ajax({
   url: 'https://cdn.jsdelivr.net/gh/chattahippie/slack-night-mode@fcafbca8be2a720410c6b3988f280fa09ef8fca0/css/raw/variants/solarized-dark.css',
   success: function(css) {
     $("<style></style>").appendTo('head').html(css);
   }
 });
});

It is a mix from the original repo and this fork.

Ubuntu Server 18.04 LTS upgrade issues and their solutions

Mon, 28 Jan 2019 00:00:00 +0000

I run KVM, PostgreSQL, Kubernetes and the ELK Stack on Ubuntu at home. These computers were running Ubuntu Server 16.04 LTS. These are the solutions to the problems encountered during the upgrade to the latest LTS version 18.04.

Disclaimer: These solutions may not be appropriate for you. Follow them at your own risk.

Upgrade Process

The operating system prompts you to upgrade whenever you log into a computer with an outdated distribution.
After creating the mandatory backups I ran the suggested command.

do-release-upgrade

The upgrade is relatively smooth. It displays several prompts, and everything works fine until it doesn’t.

Problems during the upgrade

`/boot` partition is full

Some computers that run for years, accumulate old kernels in the /boot partition. The upgrade reaches a point where apt crashes and nothing can be installed.

This guide to cleanup the /boot partition proved invaluable.

Note: The process to remove old kernels may have to be executed multiple times. Because sudo apt-get -f install can fill the /boot partition again.

Problems after the upgrade

Local DNS configuration disappeared

Every upgraded computer lost their local DNS configuration.

cat /etc/resolv.conf should print something like this:

nameserver 192.168.1.1   # your local DNS IP
search router123456.com  # your local search domain

If that is not the case, load the DNS configuration from the new location

sudo mv /etc/resolv.conf /etc/resolv.conf.bck
sudo ln -s /run/systemd/resolve/resolv.conf /etc/resolv.conf

KVM virtual machines failed to start

Sample Error:

$ virsh start vm1
error: Failed to start domain vm1
error: the CPU is incompatible with host CPU: Host CPU does not provide required features: hle, rtm

Note: These were virtual machines that were running on that same computer before the upgrade.

This Red Hat bug report has instructions to correct the issue.

Step 1: Edit the vm:

virsh edit vm1

Step 2: Add the following features to the cpu xml node

<feature policy='disable' name='rtm'/>
<feature policy='disable' name='hle'/>

Laptop screen does not turn off after long inactivity

This used to work in the old 16.04 version.

For the new 18.04 version I took a different approach. These commands make sure the computer doesn’t suspend when the lid is closed.

sudo sed -i 's/#HandleLidSwitchDocked=ignore/HandleLidSwitchDocked=ignore/g' /etc/systemd/logind.conf
sudo sed -i 's/#HandleLidSwitch=suspend/HandleLidSwitch=ignore/g' /etc/systemd/logind.conf
sudo reboot

Deprecated PostgreSQL package

The upgrade process mentioned the installed PostgreSQL version was deprecated. And that I should try running pg_upgradecluster and reading /usr/share/doc/postgresql-common/README.Debian.gz.

Since I had a backup of my data I reinstalled PostgreSQL and everything kept working just fine.

Bottomline

Although the Ubuntu 18.04 upgrade was not free of hiccups, these proved to be relatively easy to solve. Moreover, there was enough troubleshooting documentation readily available.

Migrating large files to GitHub

Tue, 08 Jan 2019 00:00:00 +0000

GitHub does not track files larger than 100MB. I hit this limit during a recent repository migration. git lsf and bfg helped me push everything. These are the steps I took to overcome this limitation.

Prerequisites

Create a backup of your repository.
Install the Git Large File Storage Tool.
Get the BFG Repo Cleaner.
Make sure the repository is in a clean state.

Solution

Step 1: Try to push.

git push origin master

`GH001` Error

Pushing to https://github.com/tddapps/my_secret_app.git
POST git-receive-pack (chunked)
remote: error: GH001: Large files detected. You may want to try Git Large File Storage - https://git-lfs.github.com.        
remote: error: Trace: 354c0f098fda496abeb40f7f6472b655        
remote: error: See http://git.io/iEPt8g for more information.        
remote: error: File Assets/Dependencies/libsmbclient.a is 114.65 MB; this exceeds GitHub's file size limit of 100.00 MB        
To https://github.com/tddapps/my_secret_app.git
 ! [remote rejected] change-photo-source-buttons -> change-photo-source-buttons (pre-receive hook declined)
 ! [remote rejected] critical-section -> critical-section (pre-receive hook declined)
 ! [remote rejected] master -> master (pre-receive hook declined)
 ! [remote rejected] version_1_0 -> version_1_0 (pre-receive hook declined)
 ! [remote rejected] version_1_1 -> version_1_1 (pre-receive hook declined)
error: failed to push some refs to 'https://github.com/tddapps/my_secret_app.git'
Completed with errors, see above

Write down the problematic filename. libsmbclient.a in this case.

Step 2: Remove the file from your history. Don’t worry, it will stay in your working directory.

java -jar ~/Downloads/bfg-1.13.0.jar --delete-files libsmbclient.a --no-blob-protection

Step 3: Unstage the file.

git reset

Step 4: Track the file as a Large Object.

git lfs track "libsmbclient.a"

Step 5: Stage, commit and push.

git add -A && \
  git commit -m "track large objects properly" && \
  git push origin master

Success!

Pushing to https://github.com/tddapps/my_secret_app.git
POST git-receive-pack (786 bytes)
To https://github.com/tddapps/my_secret_app.git
 = [up to date]      version_1_0 -> version_1_0
 = [up to date]      version_1_1 -> version_1_1
   8ce3374..e7fb4b8  master -> master
updating local tracking ref 'refs/remotes/github/master'
Uploading LFS objects: 100% (1/1), 120 MB | 1.4 MB/s, done
Completed successfully

Got other errors?

Repeat the steps if the error persists with other files.

Want to know more?

Serverless API Part 1: From idea to Production

Mon, 05 Nov 2018 00:00:00 +0000

I implemented a Heartbeat API to detect when any of my servers crash. I had two requirements. Cheap to operate. And disaster tolerance. Serverless technologies are designed to solve these kind of problems. Here are some lessons learned on my first AWS Lambda hands-on experiences.

This other post explains the Heartbeat Pattern in depth and the need for it. The project sources are available on GitHub.

The Ivory Tower Architect

With very little hands-on knowledge I set my product vision. A highly-scalable distributed system running on standard technologies. I decided the API should be serverless. The technology costs start very low and are driven by usage. AWS is the most mature serverless provider. And Java is a widely supported language.

This would be my first serverless project so I would keep it “simple” [grin].

After some “architecting”¹ I ended up with this design.

Single Region. AWS has very good availability and regional outages are rare. A future release could support multiple regions.
Two Kinesis streams. One to store messages as they came. Another stream for the notifications.
One DynamoDB table.
Multiple lambdas. For authentication, data persistence and analysis.

Why so complicated?

In a word: Ignorance

This initial design had too many redundant components. The project did not need to handle huge loads. Moreover, the minimum operational cost was going to be high.

DynamoDB All the Things. I researched pricing of the different AWS services. Kinesis Streams cost “as little as $0.015 per hour”. Almost $11/mo per region. So that had to go away. DynamoDB would have to solve all the storage needs. This was also my first DynamoDB project. I captured my thoughts on DynamoDB in a separate post.

Api Gateway to the Rescue. I thought I needed a lambda function for authentication and rate limiting. But Api Gateway has all that functionality built-in. One less thing to worry about.

Twilio has a nice API to send SMS. But SNS is also very easy to use. Besides, I would not have to manage a separate set of credentials.

The Serverless Framework and the serverless-dynamodb-local plugin proved to be great resources. They allow local testing, automated deployments, and infrastructure provisioning with code.

The MVP Architecture

hb-api release 0.1.0

As I started building the application I settled on the following design.

One lambda to save messages.
Another lambda to scan the table for expired hosts and send notifications.

Two functions. With built-in rate limiting, authentication, decent availability, and scalability.

BONUS: It is always the Database

hb-api release 0.2.0

Since a table scan on the DynamoDB table could have disastrous consequences. I decided to use DynamoDB’s TTL and Streams to reduce the database load. DynamoDB would expire messages automatically. Any database change would trigger a lambda. Such lambda would send notifications for deletions. This design used even less computing resources. It solved the Heartbeat Pattern problem leveraging serverless technologies to its maximum.

Summary

Building serverless applications was easier than I thought. The AWS toolbox is very rich. And the Serverless Framework a great help. With very little code I was able to build a secure, easy-to-use, fault-tolerant, scalable, and cheap, service. Yet, the serverless model is a paradigm shift. Traditional operations may no longer be possible. Eventual consistency and fault-tolerance become the new standard. Thus, hands-on experience is highly valuable. Stay tuned for Part 2 to make the application tolerant to regional outages. And share the operational costs.

The Clean Architecture book provides a better approach to architecture ↩

Book Review: TDD by Example

Thu, 01 Nov 2018 00:00:00 +0000

Good books are worth re-reading. I have read TDD by Example at least three times. The book has many practical examples. It showcases the Test Driven Development(TDD) discipline in an effortless and funny style. This book has advanced my career like no other.

The Author

Kent Beck was one of the creators of the Agile Manifesto. He is the creator of the Test Driven Development discipline. He has also collaborated in other classics such as Extreme Programming Explained and Refactoring.

Part I and II: Examples

The book solves traditional computing problems with a Test Driven approach. These two sections showcase how to practice the discipline to achieve optimal results. Using static or dynamic languages. Code analysis tools reveal the astounding quality of the produced code. The examples show how the tests can drive the design into more expressive and maintainable solutions.

Stress

The book mentions stress many times. Do you recognize any of these?

Scenario 1: “We’re in a hurry we don’t have time to write tests”. Hurried changes without tests produce bad software. Bad software produces more stress.
Mitigation: Write more tests. More tests increase the quality of the software. Good software is easy to change and has almost no defects.

Scenario 2: “This design will cover all the requirements. It is complex. But perfect.” Three weeks later. “What do you mean requirements changed? How do I fit that into what I have?”
Mitigation: Software always changes. Requirements change. Embrace change. Practice Test Driven Development and change will be easier.

Scenario 3: “I have to do these 20 things for this feature. Internationalization?! Now I have to do 21”. Restroom break. “Where was I in the 15 things to do?”
Mitigation: Write down all the things to do. Add new tasks to the list as they come. Work through that list one step at a time. Nothing will ever be forgotten.

The Test Driven Development discipline is about focusing on one task at a time. The book shows how anything else should be written down in a list and dealt with later. This practice reduces stress levels and increases productivity.

Part III: Patterns

The third part of the book is dedicated to design and testing patterns. How to write and design tests. The need for isolated, independent and easy to comprehend tests. How to leverage tests to understand the behavior of dependencies.

Rapid Feedback

An important benefit of the TDD discipline is the shortening of feedback cycles. Rapid feedback lets you know when you’re moving in the right direction. Large changes need decomposition to achieve rapid feedback. This practice enhances the design.

Refactoring

Design is rarely created in a perfect way. It needs to evolve through countless changes. TDD empowers the constant refactoring flow mandated by changing requirements. The book showcases several refactoring patterns. It also covers the need to write tests before refactoring untested code. I recommend Refactoring -by Martin Fowler- for a deep dive on the topic.

Mastering TDD

The final chapter contains invaluable information for those learning the discipline.

What don’t you have to test?
How do you know if you have good tests?
How much feedback do you need?
When should you delete tests?
Can you test drive large systems?
Can you drive development with application-level tests?
How do you switch to TDD midstream?
Who is TDD intended for?
Is TDD sensitive to initial conditions?
How does TDD relate to patterns?
Why does TDD work?
How does TDD relate to the practices of Extreme Programming?

Should you read it?

Definitely. TDD by Example will not only help you become a better software engineer. The book explains tools to manage any type of work.

Book Review: Clean Architecture

Wed, 31 Oct 2018 00:00:00 +0000

Clean Architecture can be described in a word: thorough. The book explains the SOLID principles in detail. It walks through the evolution of programming paradigms. How they took away features while maintaining productivity. The book revolves around the idea that business rules are the most important application component. And they should not depend on external details. Full of practical examples. The final chapter on itself -written by Simon Brown- makes the book worth reading. In summary, Clean Architecture feels like a nice complement to Clean Code and Domain Driven Design.

The Author

I have a deep admiration for Robert C. Martin (Uncle Bob). His advice on Clean Code and the Clean Coders Videos has significantly advanced my career. Familiar readers of his work will feel like they have already read the book. I would go even further and say it could fit in two thirds of the pages.

The Stories

The author wouldn’t be himself without his stories of ancient computers and perforated cards. This book is no exception. It even has a dedicated appendix for them. Although some of the metaphors and examples may no longer apply. They certainly help explain the book lessons. And provide invaluable practical examples of what otherwise be a very abstract topic.

Architecture

This book challenges the foundational beliefs of many software architects.

The goal of software architecture is to minimize the human resources required to build and maintain the required system

And yet, this definition of architecture is a great tool to gauge the quality of a project. I particularly liked the explanations around the increased costs of lines of code. The book showcases several architecture quality measurement tools. It also describes testing patterns and their effects on an architecture. e.g. the Humble Object pattern.

Details

The book stresses the need to decouple architecture from details. It covers how GUIs, databases, and frameworks are details. And as such, should not be coupled.

I agree with that guideline. I’ve worked on several projects with high coupling. Where a button change required a database schema modification.

Should you read it?

Clean Architecture has good guidelines to design maintainable software. This book can be a good introduction for the SOLID principles. At worst it is a history of computer evolution. Definitely, worth the read.

Book Review: Rich Dad Poor Dad

Fri, 26 Oct 2018 00:00:00 +0000

Rich Dad Poor Dad is a popular personal finance book. I don’t understand why. The entire book can be summarized in one page. It duplicates entire sections. The book provides no real advice on how to become rich. It even gives outright bad financial advice. Don’t waste your time. Read A Random Walk down Wall Street and The Little Book of Common Sense Investing instead.

Good

Highlights the need for financial education.
Spend money on cash producing assets.
Don’t spend money on liabilities.

Bad

Erodes the value of traditional education.
Gives very bad advice on paying debts.
Does not provide clear tools for financial independence.

The Heartbeat Pattern

Wed, 24 Oct 2018 00:00:00 +0000

Which computers in a global network are offline?

Restaurant chains rely on thousands of computers to conduct business. Think of every credit card machine in every McDonald’s in the world. Any of those computers could stop working without warning. Connectivity outages, power losses and hardware failures are common problems. Knowing when a machine disappears is not easy. Headquarters cannot ping machines through thousands of different private networks. In-store personnel may not notice a lack of connectivity for hours. Failures can even happen when the store is empty with no human around.

How do scientists know black holes exist?

Black holes are unobservable -by definition-. And yet, scientists can know their location. ¹

Tracking transactions

Computers in a global network are like black holes. They cannot be observed. But their effects can be measured.

A computer processing sales is online. The system receiving those transactions can know which computers are online.

Are computers without recent transactions offline?

A lack of transactions does not mean there is a problem. Some markets may have low sales periods. Stores in different timezones have different activity hours. Some retailers may process a very small daily volume.

Heart beats

An average human heart beats approximately 60 times per minute. A lack of heartbeats indicates a serious health problem. That is why ICUs watch them closely.

Heartbeat Pattern

What if every computer in the network sends a small test message every minute?

A system receiving these messages can know which computers are online.
An absence of periodic messages indicates a problem.
Alerts can be sent. And the appropriate personnel can react. Sometimes before there is any business impact.

Advantages

The Heartbeat Pattern solves a critical business problem. With the simplicity of a recurrent test message. It allows headquarters to assess the status of a global network. It can be the foundation to operational intelligence and proactive problem remediation.

This pattern can even be adapted to track other indicators. Complex clients can record internet speed, computer temperature, or last user interaction.

Reference Architecture

The Heartbeat pattern can be implemented with small effort.

Client

Runs recurrent curl commands to POST HTTP requests to an API. The message payload will be a unique computer identifier.

* * * * * curl -i -H "x-api-key: SECRET" -d '{"hostId": "prod-host-1"}' -X POST https://api.retailer.com/v1/hearbeat

Sample crontab file for a computer sending heartbeats

There is no need to write any custom code. Major operating systems support recurrent tasks. And curl runs on any platform.

API

Stores the unique identifiers and the current time in a single table.

Alerts Recurrent Task

Reads all the items in the table.
Considers any computer that hasn’t sent a recent message to be offline.
Triggers the appropriate alerts for each offline computer.

Read A Brief History of Time by Stephen Hawking to learn more about black holes ↩

Book Review: Creative Selection

Mon, 08 Oct 2018 00:00:00 +0000

Creative Selection describes Apple’s product development process under Scott Forstall and Steve Jobs. Written from the perspective of a software engineer. The book describes frequent product demos as they progress through the company’s ranks. This process is what Ken Kocienda calls Creative Selection. And what makes Apple products not successful accidents but deliberate successes. The demos help discuss ideas in concrete terms. And reduce the procrastination associated with imaginary problems and situations.

A software engineer

Ken Kocienda accurately depicts what being a software engineer feels like:

The pressure to meet deadlines.
How annoying bugs are.
How weeks of efforts can get scrapped.
How valuable feedback can be delivered in suboptimal ways.
How being a manager is not the logical progression of being an engineer.
The dread for meetings.
How programming cannot solve a people problem. But people can always solve programming problems.
The pride of building a widely used product.

He does a good job explaining hard engineering problems with simple metaphors.

Beware of doing things the Apple way

While Apple has its reasons to follow certain practices, chances are your company is not Apple.

Collaboration and Secrecy

Creative Selection covers with many examples the strong secrecy ambient around product development. Apple employees had multiple NDA contracts. The author mentions how collaboration is fundamental to create great products. But many examples in the book show otherwise. Product features are generally built by a single person. Individuals can only ask for help to experts in other areas of the company as long as they don’t know what the help is for.

Team size

Small teams with direct access to executives in a fairly flat organizational structure.

Planning

Whiteboard discussions feel like work, but often they’re not, since it is too difficult to talk about ideas in the abstract.

Convergence

A long QA cycle before product releases. Where the bug count does not decrease before an imminent release date.

Patents

The author name figures in several Apple patents.

Conclusion

The practices described in the book have changed over the last years¹. Yet, this book is a firsthand account of the development of products that would end up changing the world. At worst it is an interesting history read.

The author mentions this change is one of the reasons he left Apple. ↩

Thoughts on DynamoDB

Sat, 06 Oct 2018 00:00:00 +0000

I recently built a Serverless Project that used DynamoDB. Learning to use a distributed NoSQL database is never a simple task. DynamoDB makes a strong emphasis on being intentional with the data read or written. It limits almost every operation. When limits are reached, services shut down or more capacity is added at an increased cost. These are some of the most important takeaways.

Good

DynamoDB has three features that save a considerable amount of work. Entire companies have been built around tools that [poorly] solved these problems.

Global Tables

Write data in one AWS Region and the changes propagate to other regions. No need to write replication code. Consider the alternative of managing your own replication strategy.

Streams

Run custom code whenever any record changes. Such code can be unit tested and can be as decoupled from your product as you want it to be. Streams don’t impact the performance of the live database. ¹

Time to Live (TTL)

Expire old data automatically.

Scalability

The capacity to scale DynamoDB is [mostly] limited by money. Configure Auto Scaling to handle variable loads and DAX to speed up reads. No pages, no manual intervention, scalability becomes a non-event.

Bad

Performance and scalability come with a cost. Specially when reading data.

Queries

DynamoDB has no easy way to answer this question:

Which products are more expensive than $1000?

Why?: Queries need an equality comparison on the Partition key

Indexes

Indexes are nothing more than managed table clones with different keys. They cost almost the same as the original table.

Limits Everywhere

Want to know which products are more expensive than $1000? You’re gonna need a Scan to read every item in the table. This will not only cost you, but it may also take down your service.
Want to run a common bulk operation such as “Delete all accounts from the EU”? Be prepared to write code to stagger the deletions or your service may go down.
Want to read all the orders for a customer? Consider limiting how many you read.
Proper Key selection is fundamental. Or your service may go partially down during high traffic periods.

Time to Live (TTL)

The deletion of data is not too responsive [by design]. In my experience TTL deletes items ten minutes late.

Conclusion

DynamoDB is a managed globally available service constrained by cost. It has similar problems to other comparable databases. It should be used under the right circumstances [like any other tool].

Streams are like SQL triggers. Only better [grin] ↩

Development Team Evaluation

Thu, 31 May 2018 00:00:00 +0000

Development teams have similarities and differences. These are my questions to gauge the maturity of a team based on certain parameters: Agility, Code Quality, Operational Integrity. Most of the questions don’t have a single correct answer but a spectrum of possible answers. While far from being comprehensive this test can be useful when comparing potential employers and clients.

Questions

What is your source control branching strategy?
How do you know what is your team working on?
- What needs to be done?
- What has already been completed?
How do you assign new work?
How long does it take you to go from an idea into something your customers can use?
How frequently do you do deployments?
How many bugs have you had in the last quarter?
- How many of those were regressions?
What sort of testing do you do?
When was the last time you reviewed the vulnerabilities of your system?
Do you know whom to contact in case there’s a problem?
When was your last production incident?
- Did you resolve it?
- How long did it took to get resolved?

Development Team Evaluation Explained

Thu, 31 May 2018 00:00:00 +0000

Explanation for the Development Team Evaluation questions.

What is your source control branching strategy?

This is the first question for a very good reason. If the answer is “we don’t have source control” you better run for your life.
Widely adopted patterns such as feature branches or single branch development are generally a sign of maturity.
A team struggling with their branching strategy is clearly in need of technical expertise.

How do you know what is your team working on?

What needs to be done?
What has already been completed?

Mature teams actively track their work -generally with an Issue Tracking software-.
This practice increases productivity. Problems arise when work is not tracked: features are forgotten, employees burn out, chaos reigns.

How do you assign new work?

Sharing is caring. Agile teams with strong engineering practices share responsibilities across their members. No single engineer is responsible for a single product. Multiple engineers collaborate on the same task. This collaboration reduces silos and increases the Bus Factor.

How long does it take you to go from an idea into something your customers can use?

There is not a right answer here. Feature sizes can greatly vary even on the same team.
Shorter cycle times indicate good organizational health.

How frequently do you do deployments?

Frequent deployments indicate a high level of confidence in the release and monitoring process.

How many bugs have you had in the last quarter?

How many of those were regressions?

A mature team will rarely have bugs and NEVER will have regressions.
The absence of bugs indicates strong engineering practices.

What sort of testing do you do?

There is no correct answer for this question either. Mature teams will rely heavily on automated unit tests.
Avoid places that see no value in tests.

When was the last time you reviewed the vulnerabilities of your system?

Security is important for any product in any organization. Mature teams understand this and incorporate security oriented practices into their software development lifecycle.

Do you know whom to contact in case there’s a problem?

A team with strong Operational Integrity will have clearly defined policies and SLAs around incident responses.

When was your last production incident?

Did you resolve it?
How long did it took to get resolved?

A mature team will rarely have production incidents. If there is any sort of problem it will get resolved immediately.

Book Review: The Non-Designers's Design Book

Tue, 06 Mar 2018 00:00:00 +0000

Have you ever looked at a flier and felt that it was ugly? Or at an Apple billboard and felt that it was pretty? Do you feel like you can differentiate a good design from a bad one without knowing why? Then this book is for you.

The Non Designer’s Design Book is structured around the premise that once you can identify a handful of design elements you’ll be able to determine what is wrong with a design, and consequently correct it.

Robin Williams did a great job conveying design expertise with the printed edition. The paper quality, the typography the colors, are just superb. How do I know? I read the book. [Grin].

The book teaches through hundreds of good and bad design examples. That being said, the author recognizes it is far from being a substitute to four years of design school.

Part 1: Design principles

Truly eye opening. This first part of the book covers in great detail the basic design principles: Contrast, Repetition, Alignment, Proximity, and Color. This section is sprinkled with multiple quizzes.

Part 2: Types

The author can’t deny her passion for typefaces. She covers in great detail the Categories and Relationships between typefaces. Practical examples of what to use when and where are abundant. As well as an extensive typeface reference.

Case study: tddapps.com redesign

Being a software engineer by trade, my design expertise is limited to install Bootstrap. After reading The Non Designer’s Design Book I was able to redesign this blog with minimal effort.

Before

After

Not impressed with the changes?

Alignment: Everything in the updated version is aligned to the left. This makes the design dramatically better.

Contrast: Article titles are clearly distinguishable. They have a different font size and color from the article summaries. The website name can be clearly differentiated as well as the newsletter signup form.

Proximity: Although there is a considerable amount of whitespace between articles, the summaries are close to their respective titles.

Repetition: Every article title has the same # prefix.

Summary

Design is everywhere. Being able to identify good and bad designs can be a powerful skill for anyone. The Non Designer’s Design Book is a concise and easy-to-read book with many practical lessons.

Seamlessly debug Entity Framework Core SQL commands

Tue, 06 Mar 2018 00:00:00 +0000

Entity Framework Core (EF Core) is a lightweight ORM for .NET Core. It is a complete rewrite that maintains most of the functionality of Entity Framework. EF Core brings some exciting new features such as multi-platform support. And it also misses capabilities such as the ability to seamlessly debug the generated SQL. The EF Core team is tracking this issue. In the meantime, here’s my solution to the problem.

TL;DR: Use my DebugEFCore nuget package

Before: Entity Framework

Entity Framework provides a convenient way to debug the database commands being executed.

using (var context = new SampleContext()) 
{ 
    context.Database.Log = Console.Write;
}

After: EF Core

I created the DebugEFCore nuget package to achieve the same behavior.

using DebugEFCore;

public class SampleContext : DbContext
{
  protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
  {
    var debugLoggingEnabled = true;
    optionsBuilder.EnableLogging(debugLoggingEnabled);
  }
}

How does it work?

EF Core has some built-in capabilities around logging that potentially allow the debugging of the executed SQL commands. This debugging doesn’t come without a fair share of effort and knowledge. The DebugEFCore nuget abstracts away the complexities of dealing with the EF Core internals.

Existing EF Core infrastructure

The DbContext class provides a virtual OnConfiguring method that gets invoked for every context instance.
The OnConfiguring method receives a DbContextOptionsBuilder parameter.
The DbContextOptionsBuilder class has a UseLoggerFactory(ILoggerFactory) method to configure the DbContext logging.

`DebugEFCore` internals

Create an implementation of the ILoggerFactory interface. This implementation will proxy every log message it receives to a log4net logger. Detailed Source Code.
The nuget adds an extension method EnableLogging to the DbContextOptionsBuilder class. This method will call the DbContextOptionsBuilder.UseLoggerFactory only when logging is necessary.

FAQs

Why debug the SQL code an ORM produces?

ORMs abstract away many of the complexities of dealing with databases. They streamline the development process and help engineers focus on the task at hand.

Many software projects work for years without encountering performance problems. Others are not so lucky: processes start to take longer, applications crash randomly, the same code doesn’t work in different environments.

The database is usually a smoking gun during performance issues. The problem with ORMs is that they abstract away the database interactions. A minor change in the ORM usage could ripple into major database performance issues. In times like these is necessary to debug the generated SQL code in order to effectively tweak the ORM usage.

Are ORMs bad?

No. They are just tools. Martin Fowler’s views on the subject.

Should I write my own SQL commands?

No. Replicating an ORM feature set is a gargantuan task. Similar Stack Overflow question.

Book Review: Agile Web Development with Rails 5.1

Sun, 04 Mar 2018 00:00:00 +0000

Ruby on Rails (Rails) is a web MVC framework used by a diverse audience ranging from students to large companies with hundreds of employees. Agile Web Development with Rails 5.1 does a great job closing any knowledge gap for those with a Rails application.

Rails is strongly opinionated. It favors a Convention over Configuration approach to most problems. I think that is why it provokes very different emotions on those who use it. Some people love it while others hate it.

I’ve noticed very few Rails developers take the time to truly understand how a Rails application works. These lack of understanding can be disastrous for web projects: wiped out databases, unmaintainable code, new team members can’t get the app running.

Agile Web Development with Rails 5.1 is a great book for anybody with a Rails app. It showcases the majority of the framework components in a very easy to read style.

Part 1: Starting with Rails

Covers how to create a new application, how to deploy it, as well as some Ruby introduction.

Part 2: Building a web store from scratch

My favorite part of the book. With very little code, the authors built a fully functional web store. They even wrote unit tests for every component.

This part of the book teaches how to name things and where to place them to leverage the full potential of the framework.

Part 3: Deep dive into Rails

In depth coverage of topics such as Active Record, migrations, routing, replacing Rails components.

Lazy properties without compromises in C#

Thu, 01 Mar 2018 00:00:00 +0000

The Lazy pattern is very helpful and widely adopted. However, most of the times its usage comes with compromises: the code looks ugly, it is not thread-safe, locks everywhere. The book Functional Programming in Java contains a great implementation of the Lazy pattern. This post is my attempt to explain the C# implementation.

TL;DR: Use my LazyPropertyHelper nuget package for lazy properties. Read Functional Programming in Java.

These are two common attempts to implement the Lazy pattern. Each has its own benefits and drawbacks.

All code samples can be found here.

Attempt 1: Being Lazy and naive

Throughout my career I’ve written code like this. The code performs an expensive computation only when needed and caches the result to prevent unnecessary computations.

public class MyServiceNaive
{
  private ExpensiveObject _expensiveLoad;

  public ExpensiveObject ExpensiveLoad
  {
    get
    {
      if (_expensiveLoad == null)
      {
        _expensiveLoad = new ExpensiveObject();
      }

      return _expensiveLoad;
    }
  }
}

This code can be subject to weird race-conditions in a multi-threaded environment.

Attempt 2: Being Lazy and inefficient

The next approach is much better in terms of thread-safety. The expensive computation is executed only once.

public class MyLockedService
{
  private object _criticalSection = new object();

  private ExpensiveObject _expensiveLoad;

  public ExpensiveObject ExpensiveLoad
  {
    get
    {
      lock (_criticalSection)
      {
        if (_expensiveLoad == null)
        {
          _expensiveLoad = new ExpensiveObject();
        }
      }

      return _expensiveLoad;
    }
  }
}

This code has one major drawback: the lock gets acquired on every read. This behavior can greatly impact performance of multi-threaded applications.

Lazy properties without compromises

Wouldn’t it be nice if the two previous solutions could be merged? Implement a thread-safe lazy pattern where the lock is acquired only once, and subsequent reads are fast.

public class MyAwesomeService
{
  private object _criticalSection = new object();
  private Func<ExpensiveObject> _expensiveLoadReader;

  public MyAwesomeService() => _expensiveLoadReader = createAndCacheExpensiveLoad;

  public ExpensiveObject ExpensiveLoad => _expensiveLoadReader();

  private class ExpensiveObjectFactory
  {
    private readonly ExpensiveObject _cachedResult = new ExpensiveObject();

    public ExpensiveObject Build() => _cachedResult;
  }
  
  private ExpensiveObject createAndCacheExpensiveLoad()
  {
    lock (_criticalSection)
    {
      if (_expensiveLoadReader == createAndCacheExpensiveLoad)
      {
        _expensiveLoadReader = new ExpensiveObjectFactory().Build;
      }
    }
    
    return _expensiveLoadReader();
  }
}

That is certainly a lot of code. Let’s break it down:

ExpensiveLoad is a property that returns whatever _expensiveLoadReader returns
_expensiveLoadReader will call createAndCacheExpensiveLoad by default
createAndCacheExpensiveLoad will replace _expensiveLoadReader the first time it gets called. This replacement will be thread-safe
Subsequent calls to _expensiveLoadReader will return whatever ExpensiveObjectFactory.Build returns. These calls don’t need locks because they’re not mutating anything
ExpensiveObjectFactory.Build returns _cachedResult
_cachedResult is only calculated once

This solution is certainly effective. And very complex too. Who would want to write all that code for a Lazy property?

`LazyPropertyHelper` nuget to the rescue

All problems in computer science can be solved by another level of indirection. David Wheeler.

I created the LazyPropertyHelper nuget to encapsulate the solution in a reusable way. It is open source, free, and ready to use.

public class MyServiceThatUsesLazyPropertyHelperNuget
{
  private readonly Func<ExpensiveObject> _expensiveLoadReader = 
    LazyProperty.Create(() => new ExpensiveObject());
  public ExpensiveObject ExpensiveLoad => _expensiveLoadReader();
}

The service code gets considerably simplified. Let’s break it down:

ExpensiveLoad is still a property that returns whatever _expensiveLoadReader returns
_expensiveLoadReader will create a new ExpensiveObject() the first time it gets called. It will cache the result in a thread-safe manner
Subsequent calls to _expensiveLoadReader will always return the cached object without the need for locks

Summary

The complexities of implementing the Lazy pattern in a thread-safe an efficient manner can be abstracted out into a reusable library.

Book Review: Functional Programming in Java

Tue, 27 Feb 2018 00:00:00 +0000

Functional Programming in Java is an excellent read like almost every other book from the Pragmatic Bookshelf. It is written in a simple and concise style that facilitates reading. The book is sprinkled with references to other great books and dozens of code examples.

Functional Programming in Java can be a great introduction for those uninitiated with the functional programming paradigm and its benefits. It can also bring great value to more seasoned developers with experience in the platform. It showcases many of the Java 8 additions around lambdas and the Stream API. The book even includes an appendix with the existing Functional Interfaces in the JDK.

The book also covers platform independent patterns that can help virtually every developer such as Memoization, TCO, Thread-Safe lazy initialization, and many more. Moreover, many of the patterns showcased in the book can be seamlessly ported to other platforms.

Go read it now

Thoughts on side projects

Tue, 17 Oct 2017 00:00:00 +0000

My first paid job was writing Windows Applications in C#. My work computer was a clunky HP -originally intended to be used as a server-. During my nights and weekends I started learning Cocoa on a Dell laptop with no battery or display, connected to a 15” CRT monitor, running a Hackintosh version of OS X. A few years passed by… I bought my first Mac, my first iPhone, and my company needed to develop some iOS apps. Guess what iOS developer with several years of experience was readily available?

Years later I was working almost exclusively writing WPF Windows applications. I got interested in rich client-side web apps because remotely updating my clients was suboptimal. I started learning a little bit of Knockout and Angular on my nights and weekends. Guess whom got hired to work exclusively writing Angular web applications?

I can share other similar examples but I don’t want to bore you.

Whatever technology you learn, whatever activity you engage with: blogging, giving presentations, etc, is actively benefiting your employer. People rarely unlearn things they learn on their free time. I’m going to take it even a step further, that lunch you ate at your desk so you could code on that Kotlin project… that is also good for your employer.

Employees

Learn new things, have personal projects, share your knowledge, go to meetups, contribute to open source projects. Well rounded professionals know many different technologies, even if they are not related to their day-to-day jobs.

Employers

Encourage your employees to learn for the sake of learning. You’ll reap the benefits. Encourage your employees to have side projects, this will create more mature employees that can schedule their time better.

Do you prefer to be this guy?

Or this guy

But what if?

My employees work on their personal projects all the time:

Is their work done?: Yes. Great, maybe that employee needs bigger challenges. Give it to them before they go somewhere else.
No. Have an honest conversation and share this information with the employee. Take corrective actions if the problem persists.

My employees are preparing for their next interview:

Nobody can prevent an employee from wanting to go somewhere else. Figure out why would they want to leave and solve their problem.

This company have enough work for three lifetimes, they should be working for me 100% of their time:

Are the things to do publicly and readily available?
Are your employees aware that they can work 100% of their time?
Are you willing to pay for somebody directly working for you 100% of their time?
Do you need to hire more people?
Do you need to work smarter rather than harder?
Are you solving the right problems?
I’ve had countless situations where a good night of sleep solved my problem better than an entire afternoon glued to a keyboard. Some context switching can be beneficial sometimes.

I want to know what everybody is doing all the time in a spreadsheet

Ubuntu installation on KVM

Thu, 30 Mar 2017 00:00:00 +0000

Ubuntu is a very popular Linux distribution [citation_needed] and KVM its default virtualization infrastructure. Unfortunately, creating Ubuntu virtual machines for KVM is not a simple process. I built a tool to automate parts of this process and here’s how to use it.

Prerequitistes

Installation

git clone --depth 1 https://github.com/camilin87/kvm-automation.git && cd kvm-automation

Create the virtual machine

This command creates an Ubuntu virtual machine named vm1 in the ~/vms folder. The new machine will have two CPUs, four GB of RAM, seven GB of hard drive, and will use the bridge network adapter br0

ruby vm_task.rb UbuntuVmCreationTask                     \
    --path ~/vms/ --name vm1                             \
    --cpu 2 --ram 4096 --hdd 7 --br br0                  \
    --os-variant "ubuntu16.04"                           \
    --img ~/vm-templates/ubuntu-16.04.2-server-amd64.iso \
    --vnc-port "5921"

Complete the installation

Fire off your favorite VNC client and connect to your KVM host on port 5921

kvm-automation caveats

Supports only CoreOS and Ubuntu
Designed to use exclusively bridge network adapters using DHCP
Does not automate the entire Ubuntu machine provisioning
Deletes pre-existing machines with the specified name
The VNC traffic is not encrypted and it is available on all the network interfaces of the host
The new machine will not be accessible over VNC if the port is already being used by another machine

CoreOS installation on KVM

Tue, 28 Mar 2017 00:00:00 +0000

CoreOS is a lightweight Linux distribution designed specifically to run containers. KVM is the default virtualization infrastructure for Ubuntu. Unfortunately, creating a CoreOS virtual machine for KVM is not a simple process. I built a tool to automate this process and here’s how to use it.

Prerequitistes

Installation

git clone --depth 1 https://github.com/camilin87/kvm-automation.git && cd kvm-automation

Create the virtual machine

This command creates a CoreOS virtual machine named vm1 in the ~/vms folder. The new machine will have two CPUs, four GB of RAM, seven GB of hard drive, and will use the bridge network adapter br0

ruby vm_task.rb CoreOsVmCreationTask         \
    --path ~/vms/ --name vm1 --dwnld true    \
    --cpu 2 --ram 4096 --hdd 7 --br br0      \
    --key ~/.ssh/id_rsa.pub

How does it work?

kvm-automation generates a hard drive after downloading the latest Stable CoreOS QEMU Image. It then builds a cloud-config.yml out of the public RSA key and the machine name. The tool then uses this information to create a KVM virtual machine that will autostart on system boot.

kvm-automation caveats

Supports only CoreOS and Ubuntu
Designed to use exclusively bridge network adapters using DHCP
Does not allow guest customization through the cloud-config.yml
Deletes pre-existing machines with the specified name

How to run any container on a schedule

Sat, 18 Feb 2017 00:00:00 +0000

Configuring a recurring task on a Docker container requires an innumerable number of steps. Moreover, an image designed to “compute an hourly database health report” can rarely be reused to “check the api server status every three minutes”. camilin87/docker-cron is a Docker image that interacts with the Docker Engine Api to create and run containers on a schedule. The following examples describe its usage and inner workings.

Task 1: Ping `www.google.com`

This snippet creates a new container named urlStatusChecker based on the dperson/smokeping:latest image. It will run the ping www.google.com -c 1 command on startup.

docker run --rm --name urlStatusChecker \
  dperson/smokeping:latest              \
  ping www.google.com -c 1

Task 1.1: Ping `www.google.com` every three minutes

The following snippet also creates a urlStatusChecker container to ping www.google.com. However, this new container will get recreated and executed every three minutes.

cat > containerInfo.json <<EOL
{
    "Image": "dperson/smokeping:latest",
    "Name": "urlStatusChecker",
    "Cmd": [
        "ping",
        "www.google.com",
        "-c",
        "1"
    ]
}
EOL

docker run --rm -d --name docker-cron                         \
  -v /var/run/docker.sock:/var/run/docker.sock                \
  --env TASK_SCHEDULE='*/3 * * * *'                           \
  -v $PWD/containerInfo.json:/usr/src/containerInfo.json      \
  camilin87/docker-cron

How does it work?

The code from Task 1.1 is running a container named docker-cron based on the camilin87/docker-cron image. That container will in turn create a new urlStatusChecker container based on the dperson/smokeping:latest every three minutes.

TASK_SCHEDULE: an environment variable to configure the frequency

containerInfo.json: a file that contains the detailed specification of the task to run. It can describe any aspect of the spawned container -volumes, networks, cmd, …- as long as it is supported on the Docker Engine Api. The camilin87/docker-cron image is built to POST this file to the /containers/create Docker Engine Api method.

/var/run/docker.sock: Is the socket where the Docker Daemon is listening to. It needs to be mounted as a volume for the docker-cron container to interact with the Docker Engine Api.

Things we didn’t have to do

Let’s analyze some tasks we didn’t have to do to ping www.google.com every three minutes:

We did not have to create a new Docker image. We were able to reuse the same dperson/smokeping:latest image we were using before.
We did not have to install nor configure cron nor any other dependency.
We did not have to configure a new cron task and all the complexities it involves with configuration files and log rotation.
We don’t have to worry about memory leaks or performance degradation over time. Simply because a brand new instance of dperson/smokeping:latest is created and destroyed every three minutes.

Task 2: Parameterize the Ping url and count

The following task is very similar to Task 1. However, in Task 2 the url and the number of requests can be parameterized through the PING_URL and PING_COUNT environment variables.

docker run --rm --name urlStatusChecker     \
  --env PING_URL="www.facebook.com"         \
  --env PING_COUNT=1                        \
  dperson/smokeping:latest                  \
  sh -c "ping \$PING_URL -c \$PING_COUNT"

Task 2.1: Ping `www.facebook.com` every three minutes

This would be the equivalent parameterized scheduled Ping.

cat > containerInfo.json <<EOL
{
    "Image": "dperson/smokeping:latest",
    "Name": "urlStatusChecker",
    "Cmd": [
        "sh",
        "-c",
        "ping \$PING_URL -c \$PING_COUNT"
    ]
}
EOL

docker run --rm -d --name docker-cron                         \
  -v /var/run/docker.sock:/var/run/docker.sock                \
  --env TASK_SCHEDULE='*/3 * * * *'                           \
  --env PING_URL='www.facebook.com'                           \
  --env PING_COUNT=1                                          \
  --env COPY_ENV_VARS='PING_URL,PING_COUNT'                   \
  -v $PWD/containerInfo.json:/usr/src/containerInfo.json      \
  camilin87/docker-cron

How does it work?

The camilin87/docker-cron image copies environment variables from the docker-cron container into the urlStatusChecker container. However, -for security reasons- it does not copy all the environment variables. It only copies those environment variables specified in COPY_ENV_VARS.

Once more camilin87/docker-cron is leveraging existing functionality of the /containers/create Docker Engine Api method. It is injecting the environment variables specified in COPY_ENV_VARS into the Env field of the Api request.

Why so much hassle with environment variables?

Environment variables have traditionally been the foundation to provide non-persistent configuration to containers. Any decent container scheduler should provide a secure way to deal with them.

Bonus: Docker Swarm support

Docker Swarm provides some nice monitoring capabilities out of the box. The following snippet has the same functionality as the one from Task 2.

docker service create                                                                       \
  --replicas 1                                                                              \
  --name docker-cron                                                                        \
  --env TASK_SCHEDULE='*/3 * * * *'                                                         \
  --env PING_URL='www.facebook.com'                                                         \
  --env PING_COUNT=1                                                                        \
  --env COPY_ENV_VARS='PING_URL,PING_COUNT'                                                 \
  --mount type=bind,source=/var/run/docker.sock,target=/var/run/docker.sock                 \
  --mount type=bind,source=$PWD/containerInfo.json,destination=/usr/src/containerInfo.json  \
  camilin87/docker-cron

Additional resources

Visit The Cron Format to learn more about cron job frequencies
All the source can be found here
This readme file contains additional usage and configuration details
camilin87/docker-cron uses the Docker Engine Api 1.24 for compatibility reasons with Docker 1.12

CoreOS baremetal server installation

Mon, 16 Jan 2017 00:00:00 +0000

CoreOS is a lightweight Linux distribution designed specifically to run containers. Unfortunately its installation on baremetal servers is not as streamlined as other Linux distributions. These are the steps I took to install CoreOS on a couple of physical machines.

Prerequisites

Download and burn a CoreOS ISO
Generate ssh keys to log into the boxes

Generate a cloud-config.yml file

 cat > cloud-config.yaml <<EOL
 #cloud-config
 ssh-authorized-keys:
     - `cat id_rsa.pub`
 EOL

This cloud-config.yaml can be reused to provision multiple computers

Copy the cloud-config.yaml file to a flash drive
Additional cloud-config.yaml help

Prepare the installation

Connect the computer to the LAN
Boot up from the CD
Wait until a command prompt appears
More information on the CoreOS installation to disk

Copy the `cloud-config.yaml` into the server

Plug in the flash drive with the cloud-config.yaml
List the physical disks
```
 lsblk
```
Determine what is the first partition of the USB drive. In this example we’ll use /dev/sdb1 where /dev/sdb is the physical device

Copy the cloud-config.yaml

 sudo su - root
 mkdir ~/flash
 mount -o ro /dev/sdb1 ~/flash
 cp ~/flash/cloud-config.yaml ~/
 umount /dev/sdb1
 rm -Rf ~/flash

Unplug the flash drive

Run the installation

coreos-install -d /dev/sda -c ~/cloud-config.yaml
reboot

At this point the server should boot up using DHCP. Its console will display its IP and MAC address.

Remote into the server

ssh into the box using your private RSA key

ssh -i id_rsa core@192.168.1.80

Success!

BONUS: Configure the network to use a static IP

List the Network Interfaces
```
 ifconfig
```
Determine the name of the Ethernet Adapter. In this example we’ll use eno1

Execute the following commands using the appropriate values for your network

 sudo su - root

 cat > /etc/systemd/network/static.network <<EOL
 [Match]
 Name=eno1

 [Network]
 Address=192.168.1.10/24
 Gateway=192.168.1.1
 DNS=8.8.8.8
 EOL

 reboot

More information on CoreOS network configuration

At this point the server should boot up using the new IP.

How to use your Keybase key for ssh

Sun, 15 Jan 2017 00:00:00 +0000

Extracting an RSA key out of Keybase is not straightforward for security beginners. These are the steps I took to automate the process.

The problem

Manually managing usernames and passwords for ssh into servers can be a difficult task. All those different passwords need to be backed up, synchronized across multiple computers, and typed every single time. Although password managers mitigate the credentials problem there’s a better mechanism to remote into boxes without typing passwords. The process involves generating public and private RSA key files.

These RSA keys need to be backed up and shared across different computers. While some use Dropbox to keep their keys synchronized others use Keybase. It is an online service to store PGP keys and map them to public identities such as Twitter accounts and domain registrations.

Prerequisites

Have a Keybase account.
Have a working Docker installation

Solution

The following commands generate ssh keys out of the Keybase PGP keys

TMP_KEY_STORAGE=~/devroot/keys
mkdir -p $TMP_KEY_STORAGE
docker run -it --rm -v $TMP_KEY_STORAGE:/home ubuntu:16.04 /bin/bash

apt-get update -y
apt-get install curl monkeysphere -y
curl -O https://prerelease.keybase.io/keybase_amd64.deb
dpkg -i keybase_amd64.deb
apt-get install -f -y
rm keybase_amd64.deb

groupadd -g 1001 user_data
useradd --create-home -g user_data -u 1001 user_data
su user_data
cd /home/user_data

keybase login

# Exporting your Keybase public key to keybase.public.key
keybase pgp export -o keybase.public.key
# Exporting your Keybase private key to keybase.private.key
keybase pgp export -s --unencrypted -o keybase.private.key

# Import your Keybase public key
gpg -q --import keybase.public.key
# Import your Keybase private key
gpg -q --allow-secret-key-import --import keybase.private.key
# The key import process produces a short hexadecimal hash
# We need to extract this hash and use it to generate the RSA key
# The hash is temporarily saved into hash.key
gpg --list-keys | grep '^pub\s*.*\/*.\s.*' | grep -oEi '\/(.*)\s' | cut -c 2- | awk '{$1=$1};1' > hash.key

ENC_KEY=`cat hash.key`
echo $ENC_KEY

# Generate the RSA private key using the hexadecimal hash
# The private key will be saved in the id_rsa file
gpg --export-options export-reset-subkey-passwd,export-minimal,no-export-attributes --export-secret-keys --no-armor $ENC_KEY | openpgp2ssh $ENC_KEY > id_rsa
# Secure the private RSA key file  
chmod 400 id_rsa
# Generate the public RSA key file  
ssh-keygen -y -f id_rsa > id_rsa.pub

# Remove all the temporary files  
rm *.key

exit
exit

chmod 400 $TMP_KEY_STORAGE/user_data/id_rsa

Bonus: Replace the default ssh key

# Delete any pre-existing key  
rm -f ~/.ssh/id_rsa.pub
rm -f ~/.ssh/id_rsa

mkdir -p ~/.ssh/

# Create symbolic links to the default  
ln -s $TMP_KEY_STORAGE/user_data/id_rsa.pub ~/.ssh/id_rsa.pub
ln -s $TMP_KEY_STORAGE/user_data/id_rsa ~/.ssh/id_rsa

Want to know more?

2016 Retrospective

Wed, 28 Dec 2016 00:00:00 +0000

These are my most dearest professional achievements and setbacks of 2016. Happy new year folks!!

5 public presentations
10 hours of in-house workshops
22 blogposts
1677 public commits
7 npm packages
11 docker hub images

Learning

21 books read
60 hours of dedicated training
- Umbraco
- Azure
- MAPR
58 personal private notes
a gazillion of podcasts listened to

Side projects

Supplement Facts Calculator
Twitter Utils
3 apps removed from the appstore

Miami Scala Meetup Presentation Notes

Wed, 14 Dec 2016 00:00:00 +0000

These are my notes of my Docker Presentation. Held at the Microsoft Innovation Center on December 14th, 2016.

Example 1

uname -a

docker run ubuntu uname -a

Example 2

ls /

docker run ubuntu ls /

Example 3: Long lived Containers

docker run -p 8080:80 seqvence/static-site

docker ps

Open localhost:8080

Example 4: Environment Variables

docker run -e AUTHOR="Miami" -p 8080:80 seqvence/static-site

Open localhost:8080

Example 5: Multiple Containers

docker run -e AUTHOR="Miami" -p 8081:80 seqvence/static-site
docker run -e AUTHOR="Downtown" -p 8082:80 seqvence/static-site

Open Site 1
Open Site 2

Example 6: Docker Compose

nano docker-compose.yaml

site1:
    image: seqvence/static-site
    environment:
        - AUTHOR=Miami
    ports:
        - 8081:80

site2:
    image: seqvence/static-site
    environment:
        - AUTHOR=Downtown
    ports:
        - 8082:80

docker-compose up

Open Site 1
Open Site 2

Cleanup docker-compose down

Example 7: Dockerfile

nano app.js

    var express = require('express')
    var app = express()

    app.get('/', function (req, res) {
      res.send('Hello World!')
    })

    app.listen(3000, function () {
      console.log('Example app listening on port 3000!')
    })

npm i express --save 
node app.js

Sample Express Site

nano Dockerfile

FROM node

RUN mkdir -p /usr/src/app

COPY app.js /usr/src/app/

RUN npm i express --save

CMD ["node", "/usr/src/app/app.js"]

docker build -t camilin87/docker-intro-node .
docker run -p 8080:3000 camilin87/docker-intro-node

Open Node Site

Example 8: Publish an Image

docker push camilin87/docker-intro-node

View Our published Image

Run it yourself docker run -p 8080:3000 camilin87/docker-intro-node

Cleanup

docker rm -f $(docker ps -a -q)

How to forward your Heroku logs to AWS Cloudwatch for free

Sun, 16 Oct 2016 00:00:00 +0000

Heroku is a great platform to run apps with little infrastructure involvement. Most of the time, you just push your code and you get a website up and running in seconds. Heroku even supports third party addons for those times when extra functionality is required. However, I could not find any addon to send my application logs to AWS Cloudwatch. The only logical solution was to build my own AWS Cloudwatch log forwarder.

How to forward the logs to AWS Cloudwatch?

The following steps are for node based apps. If you have a Heroku app made in something else drop me a line and I’ll help you set it up.

Step 1: Have a deployed NodeJs application in Heroku. How?
Step 2: Get your AWS credentials
Step 3: Setup the following Heroku environment variables

heroku config:set AWS_REGION='us-east-1'
heroku config:set AWS_ACCESS_KEY_ID='XXXXXXXXXXXXXXX'
heroku config:set AWS_SECRET_ACCESS_KEY='XXXXXXXXXXXXXXX'

Step 4: Install aws-cloudwatch-forwarder on your project

npm i aws-cloudwatch-forwarder --save

Step 5: Use it in your package.json

Change the start script in your package.json in a similar way to this:

"start": "aws-cloudwatch-forwarder 'node web.js'"

Want to see it in action?

Here’s some sample code

Book Review: So good they can't ignore you

Thu, 13 Oct 2016 00:00:00 +0000

Do you feel stuck in your job? Do you feel you don’t like any profession? You have no idea what to study? So good they can’t ignore you is the right book for you. It is a data driven book about real life examples of successful, and not so successful individuals. So good they can’t ignore you revolves around the idea that following your passion won’t get you anywhere. Instead it proposes to look at your job with a craftsman’s mindset until you become so good they can’t ignore you.

Control

Industry leaders usually have significant amounts of control over what they do. They choose what to work on and how to do things. Control is what many of us crave for. So good they can’t ignore you illustrates several individuals’ path to gain control over their careers. Hint: None of them followed their passion.

Caveat: Only pursue more control if you have evidence other people are willing to pay for it.

Career Capital

Career capital are the skills you have that are both rare and valuable and that can be used as leverage in defining your career

Making enough Career capital is the safest way to gain control over your career.

Employers will usually promote you once you have enough career capital. Beware though, most of these promotions will limit your ability to gain control over your career.

Example: The best QA technician gets promoted to QA supervisor. Suddenly she’s working twice the hours and has practically no time to spend with her family.

The Craftsman’s mindset

So good they can’t ignore you illustrates how the Craftsman’s mindset is the best way to gather Career capital. Think about your job like a craftsman would do. Practice deliberately to become better. You need to get good at what you do before you can get good work. Moreover, try to incorporate frequent feedback into the practice.

Example: Don’t write the entire book before somebody else reads it. Share each chapter with as many people as possible and incorporate their feedback.

Deliberate Practice

Talent is overrated. The majority of the leaders in their fields are so because they spend countless hours practicing. Practice outside of your comfort zone. Practice to get better.

Breakthroughs

Finally, the book describes the concept of the Adjacent Possible. How breakthroughs are only possible if they’re built incrementally on top of what’s already possible.

Summary

So good they can’t ignore you is an excellent book that will help you advance your career. If you don’t like reading books, you can always listen to it as an audiobook.

I built my own AWS Cloudwatch log forwarder as an npm package

Sun, 09 Oct 2016 00:00:00 +0000

AWS Cloudwatch is an outstanding monitoring platform. With competitive prices, it is extremely configurable and extensible. Unfortunately, monitoring resources outside of AWS is not straightforward. To solve this problem I created my own Cloudwatch log forwarder as a non-intrusive npm package.

What’s wrong with the existing log forwarding tools?

Amazon provides a CloudWatch Logs Agent to forward logs. This agent must be installed as a service running on a server. While this is easy on EC2 instances, it is impossible to do in shared environments such as Heroku.

Amazon built the CloudWatch Logs API and SDKs for developers to interact with the system. While these SDKs are relatively simple to use, they still require a fair amount of work and understanding of the API.

Third-party developers have built their own forwarding tools based on the Cloudwatch SDKs. However, they all share the same problem. The application code would have to change to explicitly log to Cloudwatch. Moreover, the majority of the javascript tools can only be used from other javascript applications.

Why I built the `aws-cloudwatch-forwarder`?

It can be used outside of AWS: The aws-cloudwatch-forwarder will process logs from anywhere.
It doesn’t need an agent installation: Built as a regular npm package, it can be used as a standalone tool such as echo or grep. Yes, you can use it to send your Heroku logs to AWS Cloudwatch.
No need to change any existing code: It can be used with existing applications. Even if they’re not built with javascript.
```
  aws-cloudwatch-forwarder 'echo "sample application"'
```

How do I use it?

Visit the aws-cloudwatch-forwarder npm page for the latest documentation

I want to see how it works

The aws-cloudwatch-forwarder is entirely open source

Docker Miami Meetup Presentation Notes

Wed, 24 Aug 2016 00:00:00 +0000

These are the notes of my Docker Miami presentation. Held at learn01.io on August 24th, 2016.

Run the logspout container

docker run -h $(hostname) \
    -e "AWS_ACCESS_KEY_ID=XXXXXXXXXXXXX" \
    -e "AWS_SECRET_ACCESS_KEY=XXXXXXXXXXX" \
    -e "LOGSPOUT=ignore" \
    --volume=/var/run/docker.sock:/tmp/docker.sock --name=logspout \
    --rm -it mdsol/logspout 'cloudwatch://us-east-1?DEBUG=1&NOEC2'

Run the heartbeat container

docker run \
    --name=fast-heartbeat \
    -e "LOGSPOUT_GROUP=docker-talk-logs" \
    camilin87/fast-heartbeat

View the logs in AWS Cloudwatch

Create a metric on the search term `Heartbeat`

aws logs put-metric-filter \
  --log-group-name "docker-talk-logs" \
  --filter-name "hb-transmission" \
  --filter-pattern '"Heartbeat"' \
  --metric-transformations \
      metricName='hb-transmission',metricNamespace=LogMetrics,metricValue=1

Configure an alert for our metric

aws cloudwatch put-metric-alarm \
    --alarm-name "hb-error" \
    --metric-name "hb-transmission" \
    --namespace LogMetrics \
    --statistic Sum \
    --period 60 \
    --threshold 10 \
    --comparison-operator LessThanThreshold \
    --evaluation-periods 1 \
    --alarm-actions arn:aws:sns:us-east-1:511794458722:NotifyMe \
    --ok-actions arn:aws:sns:us-east-1:511794458722:NotifyMe \
    --insufficient-data-actions arn:aws:sns:us-east-1:511794458722:NotifyMe

Send your Docker container logs to AWS Cloudwatch using logspout

Sat, 02 Jul 2016 00:00:00 +0000

As a general rule Docker containers print to the STDOUT anything that should be logged. This simple approach is nothing but powerful and extensive. Logspout is a lightweight container that forwards the logs of other containers running on the same server. It is built in an extensible way with support for multiple log destinations. This guide will show how to configure Logspout-Cloudwatch to forward our logs to AWS CloudWatch.¹

The instructions of this guide are based on Docker 1.9.

Step 1: Configure AWS Cloudwatch

We need an Access Key Id and a Secret Access Key that can forward logs to Cloudwatch. Here’s how to configure AWS Cloudwatch for Log Forwarders.

Step 2: Run the logspout-cloudwatch container

docker run -h $(hostname) \
    -e "AWS_ACCESS_KEY_ID=xxxxxxxxxxxxxxxx" \
    -e "AWS_SECRET_ACCESS_KEY=xxxxxxxxxxxxxxxxxxxxxx" \
    -e "LOGSPOUT=ignore" \
    --volume=/var/run/docker.sock:/tmp/docker.sock --name=logspout \
    --rm -it mdsol/logspout 'cloudwatch://us-east-1?DEBUG=1&NOEC2'

What is this?: Start and configure a container of the image mdsol/logspout. More details on the configuration can be found here.

Step 3: Run a container that generates logs

docker run \
  -e "LOGSPOUT_GROUP=sample-docker-logs" \
  ubuntu bash -c \
  'for i in {1..10}; do echo "Hi, the date is $(date)" ; sleep 1; done'

What is this?: Start a container that will log the current date ten times and send its logs to the sample-docker-logs log group.

Step 4: Review the logs

1- Open the CloudWatch sample-docker-logs group
2- Click Search Events

Your should be able to see your logs

Bonus: Docker Cloud configuration

Docker Cloud is a SaaS tool run by Docker to manage and automate container deployments. It is very powerful and yet simple to use. More info here.

1- Create and Start a logging Stack

logspout:
  image: mdsol/logspout
  deployment_strategy: every_node
  autoredeploy: true
  restart: on-failure
  volumes:
    - /var/run/docker.sock:/tmp/docker.sock
  environment:
    - AWS_ACCESS_KEY_ID=xxxxxxxxxxxxxxxx
    - AWS_SECRET_ACCESS_KEY=xxxxxxxxxxxxxxxxxxxxxx
    - LOGSPOUT=ignore
  command: 'cloudwatch://us-east-1?DEBUG=1&NOEC2'

What is this?: A Stack YAML file to configure a logspout service. Docker Cloud will deploy a mdsol/logspout container on every node under the account. Moreover, it will make sure our container gets restarted in case of a failure.

2- Create and Start hello-world Stack

helloworld:
  image: ubuntu
  target_num_containers: 1
  environment:
    - LOGSPOUT_GROUP=sample-docker-logs
  command: bash -c 'for i in {1..10}; do echo "Hi, the date is $(date)" ; sleep 1; done'

Why Cloudwatch? Cloudwatch has excellent support for alerts and autoscaling. ↩

Configure AWS Cloudwatch for Log Forwarders

Fri, 01 Jul 2016 00:00:00 +0000

AWS CloudWatch is a monitoring service to collect logs. It can be configured to accept multiple log sources. As with other AWS services Cloudwatch has detailed security and access control support. These are the steps I take to configure any log forwarder to Cloudwatch.

This guide will produce an Access Key Id and a Secret Access Key.

Configure an Access Policy

Policies are the backbone of AWS security. It is a good practice to write them as restrictive as possible. ¹

1- Open the IAM Policies section
2- Select Create Policy
3- Select Create Your Own Policy
4- Name it CloudWatchLogSender
5- Add the following text to the Policy Document section

{
    "Version": "2016-07-02",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:DescribeLogGroups",
                "logs:DescribeLogStreams",
                "logs:PutLogEvents"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

What is this?: A policy that only Allows the actions CreateLogGroup, CreateLogStream, DescribeLogGroups, DescribeLogStreams, and PutLogEvents on any resource.

Create an AWS User

Having a specialized user just to forward logs can significantly limit the impact of any attack on the account. ²

1- Open the User Management Module
2- Create a new user named CloudWatchLogSender.
Make sure to save these security credentials because this is the last time you’ll see them

3- Open the CloudWatchLogSender user details page
4- Click the Attach Policy button in the Permissions tab

5- Attach the CloudWatchLogSender policy

Your user summary should look like this

Summary

We have created the necessary security provisions to forward logs to AWS Cloudwatch from any source. Moreover, we have credentials that can be used by any forwarder compatible with Cloudwatch.

More Info on AWS policies. ↩
Restrictive access controls limit the risk of somebody using your account to mine Bitcoins or some other crazy thing. ↩

Send your Docker container logs to AWS using the log-driver

Sun, 26 Jun 2016 00:00:00 +0000

Docker supports several logging drivers to forward container logs. These are the steps to configure the AWS CloudWatch log driver to ship the hello-world container logs.

The instructions of this guide are based on Docker 1.9.

Step 1: Configure AWS Cloudwatch

We need an Access Key Id and a Secret Access Key that can forward logs to Cloudwatch. Here’s how to configure AWS Cloudwatch for Log Forwarders.

Step 2: Configure your docker daemon to use the AWS credentials

Mac OS X

These instructions are for the Docker Toolbox because Docker for Mac is still in beta.

1- Open a Docker QuickStart Terminal
2- Stop the the default docker-machine

docker-machine stop default

3- Create a new awslogs docker-machine with the AWS credentials

docker-machine create -d virtualbox \
    --engine-env AWS_REGION=us-east-1 \
    --engine-env AWS_ACCESS_KEY_ID=xxxxxxxxxxxxxxxx \
    --engine-env AWS_SECRET_ACCESS_KEY=xxxxxxxxxxxxxxxxxxxxxx \
    awslogs

Ubuntu

The following GitHub Issue details the steps to configure AWS credentials on Ubuntu.

Docker Cloud

No support for log drivers yet.

Step 3: Run the Container

docker run \
    --log-driver=awslogs \
    --log-opt awslogs-region=us-east-1 \
    --log-opt awslogs-group=sample-docker-logs \
    --log-opt awslogs-stream=hello-world-logs \
    hello-world

Step 4: Review the logs

1- Open the CloudWatch sample-docker-logs group
2- Open the hello-world-logs stream

Your should be able to see your logs

Book Review: The Ideal Team Player

Mon, 06 Jun 2016 00:00:00 +0000

As with other Lencioni books The Ideal Team Player is hard to put down once you start reading it. It emphasizes the significative advantage teamwork can bring to a company. The book revolves around the “Humble, Hungry, Smart” model. What is it and how ideal team players should follow it.

The Ideal Team Player is far from being a theory book. It provides practical applications of the “Humble, Hungry, Smart” model: interview techniques, interview questions, employee assessment and development techniques, techniques to embed the model in the culture.

The book describes several personalities I’ve encountered during my years working: the Pawn, the Bulldozer, the Charmer, the Lovable Slacker, the Skillful Politician.

My one sentence summary would be: “Don’t hire jackasses. If you keep them for too long then the non-jackasses will leave”.

Go read it now and find out for yourself.

How to run Node cron jobs inside a Docker container

Thu, 05 May 2016 00:00:00 +0000

Although there are several tutorials on how to run cronjobs inside Docker containers. They mostly focus on very simple tasks that don’t need to read environment variables. Moreover, there are node specific solutions. I didn’t like those either because they rely on the stability on a long running process. The following guide is my solution to the problem after many hours of searching through StackOverflow and good old experimentation.

TL;DR: Create your own Dockerfile with the following contents

FROM camilin87/node-cron:latest

ENV TASK_SCHEDULE='* * * * *'
COPY . /usr/src/app

The base image

Given the complexity of scheduling a node cron job, the best way I could find to abstract that out was to create my own base image.

Inputs

/usr/src/app: Directory where the executable code shall reside. The image will run npm start from this directory.
TASK_SCHEDULE: Environment variable with the cron schedule. e.g. TASK_SCHEDULE='* * * * *' configures a task that runs every minute Cron HowTo

Outputs

/var/log/cron.log: All the execution logs

How to use it?

Create your own Dockerfile based off the camilin87/node-cron image.
Make sure to copy your code into /usr/src/app. You should have a package.json file with a start script. The image will run npm start from /usr/src/app.
Specify the TASK_SCHEDULE environment variable. It can be done in the Dockerfile or in the docker run command.

Here’s a sample Dockerfile for a task that runs every minute.

FROM camilin87/node-cron:latest

ENV TASK_SCHEDULE='* * * * *'
COPY . /usr/src/app

How does it work?

Our base image is nothing more than a Dockerfile and a couple of shell scripts. Its source code is free and here’s a sample application that uses it. If you feel that something can be improved please create a pull request. Let’s deconstruct the base image step by step.

Dockerfile

FROM node:latest                                   # base this image off the official node image
                                                   # if you need a specific node version
                                                   # simply change latest for what you want
                                                   #
                                                   # more info here => https://hub.docker.com/_/node/

RUN apt-get update && apt-get install -y cron      # install cron
                                                   #
                                                   # more info => https://help.ubuntu.com/community/CronHowto
                                                   #           => https://docs.docker.com/engine/userguide/eng-image/dockerfile_best-practices/#apt-get

RUN mkdir -p /usr/src/app                          # our code will be shipped with the new container
                                                   # into the /usr/src/app directory
                                                   # this line creates the directory preemptively

COPY ./templates/crontab /tmp/crontab              # cronjobs are configured as files
                                                   # with a very specific format
                                                   # the file ./templates/crontab will be the base template
                                                   # for all of our cron jobs
                                                   # it will be explained in details later on

RUN touch /etc/cron.d/my-cron-job                  # our cron job configuration will end up in /etc/cron.d/my-cron-job
                                                   # this line creates the file preemptively

RUN chmod 0644 /etc/cron.d/my-cron-job             # give permissions to the cron job configuration file

RUN touch /var/log/cron.log                        # the logs from our job
                                                   # this line creates the log file preemptively

COPY ./templates/setupCron.sh /tmp/setupCron.sh    # to execute our cronjob we must run multiple commands
                                                   # it is not possible to do it directly from the Dockerfile
                                                   #
                                                   # the file ./templates/setupCron.sh will be
                                                   # executed when the container runs
                                                   # it will be explained in details later 
                                                   #
                                                   # more info => https://docs.docker.com/engine/reference/builder/#cmd

RUN chmod +x /tmp/setupCron.sh                     # give execution permissions to the command

CMD ["/tmp/setupCron.sh"]                          # configure the command to run when the container runs

templates/crontab

The file ./templates/crontab contains the initial template of our cron job.

 root cd /usr/src/app && npm start >> /var/log/cron.log 2>&1
# An empty line is required

Cron configuration breakdown

User: root
Working Directory: /usr/src/app
Logs: /var/log/cron.log
Command: npm start This base image is highly opinionated on that sense. Since it’s running a node cron job it assumes what the command to execute is.
Schedule: The schedule is missing from this configuration template because it will be determined when the image runs the CMD. That way it can be configurable.

templates/setupCron.sh

The most important file of our image. ./templates/setupCron.sh is the command that will be executed when our image runs. It will be responsible for making the container environment variables visible to the cron job.

#!/bin/bash
env                                           >> /tmp/.env                   # save all the environment variables
                                                                             # into /tmp/.env

cat /tmp/.env                                 >> /etc/cron.d/my-cron-job     # write the environment
                                                                             # variables into the cron job configuration

echo -n "$TASK_SCHEDULE" | cat - /tmp/crontab >> /etc/cron.d/my-cron-job     # write the TASK_SCHEDULE variable
                                                                             # into the cron job configuration
                                                                             # along with the contents 
                                                                             # of the /tmp/crontab file
                                                                             # which is nothing more than the
                                                                             # ./templates/crontab file from the image repo

cron && tail -f /var/log/cron.log                                            # run cron and append its logs
                                                                             # to the log file from our task

Why so complicated?

The biggest hurdle was making the environment variables visible to the cron job. Since cron knows nothing about the running shell the best way I could find was to include them in the crontab file.

The other caveat was that none of this variable replacing magic could be done in the Dockerfile. Why? Because most of the environment variables are being set only when the image is run. The substitution had to be done at runtime. Thus the need for a big shell script.

Death by Meeting

Wed, 04 May 2016 00:00:00 +0000

Like other Patrick Lencioni books Death by Meeting is very enjoyable and easy to read. The book is targeted at making your meetings more effective rather than abolishing them altogether.

Meeting Requirements

Death by Meeting clearly illustrates the necessity of Conflict and Contextual Structure for effective meetings.

Conflict: Meeting leaders should mine for conflict out of the participants. They should remember attendees that conflict is good and expected. Conflicts not addressed in meetings can easily become politics.

Contextual Structure: Some meetings should be focused on solving an immediate problem while others should be purely brainstorming. Do not mix meeting structures.

Suggested Meetings

Near the end of the book the author describes the four types of meetings companies should have.

Daily CheckIn: Five minutes standing up. Report current activities.
Weekly Tactical: 45 to 90 minutes. 60 seconds per participant to provide context. 5 minutes of progress review based on key business metrics. Realtime agenda derived from the progress review. Limited to address short term topics.
Monthly Strategic: Deep dive into one or two topics that will impact the business. Two hours per topic. Can be scheduled ad-hoc too for critical issues. Require proper preparation.
Quarterly Offsite Review: Analyze the business in a more complete manner. One or two days. Team and strategy review. Competitors and industry reviews. Aim to improve team unity.

Summary

Straight to the point book. Mandatory read for meeting organizers and participants.

The Goal and The Phoenix Project

Mon, 02 May 2016 00:00:00 +0000

These two books are very hard to put down. They both make an emphasis in looking at the big picture and what’s relevant for the company as opposed to individual departments.


The Phoenix Project on Amazon	The Goal on Amazon

Takeaways

It doesn’t matter how hard you work if it’s not aligned with the company’s goal
A factory where every station is working at 100% of its capacity is very inefficient
The company’s velocity is limited by the slowest link in the chain

Written in a very similar style both books favor the idea that asking questions is the only true way of teaching. They both clearly illustrate the importance of cross-team collaboration to impact the company’s bottomline.

Automating Level3 Cache Invalidation

Fri, 01 Apr 2016 00:00:00 +0000

One of the companies I’ve worked for uses Level3 as their CDN. As part of the release process for one of the products we had to manually clear the CDN cache. This was a manual multi-step process that -more often than not- was forgotten. Automation to the rescue.

The API

Fortunately, Level3 has a REST API with all sort of capabilities. They even provide some Sample Code in a variety of languages. One of this samples is a full-blown debugging tool.

Step 1: Get an API Key

In order to use the Level3 API you must first get an API Key. This needs to be done by an account administrator.

The Application

After ~~ruthlessly copying~~ reviewing the sample code provided by Level3. I was able to create a C# console application to automate the cache invalidation. It receives the following arguments:

level3_api_key
level3_api_secret
level3_invalidation_urls
level3_notification_email

All the necessary code is available in GitHub and here’s a rough description of how it works.

Octopus Integration

Octopus is a great tool to automate and manage deployment pipelines. The Level3 Cache Invalidation was crafted with Octopus in mind so that it can be invoked as part of any automated deployment. This Readme file indicates how the code can be used with Octopus.

Tribal Leadership and why you should read it

Wed, 30 Mar 2016 00:00:00 +0000

First things first. I think the title of the book is misleading. This is not a business administration book. Moreover, it is far from teaching anyone how to be a leader.

The book is a self-discovery book that gives names to common behaviors encountered throughout almost every workplace. And a very good one to be fair.

The book answers Why does it feel like:

Every company has the same issues
It is very hard to fix broken processes
Perennial failure in spite of being constantly working
The people around can’t be as good

Tribal Leadership categorizes these and several other behaviors into a set of five simple stages. Not only that, it provides tools to measure and increase the throughput of your organization.

I don’t want to give too much away. Go read it if you’re interested in learning more about yourself and your organization dynamics.

Demystifying the Open Source

Mon, 28 Mar 2016 00:00:00 +0000

A lot has been said about open source technologies. Many Fortune 500 businesses run open source software. However, many misconceptions about the open source still lurk around the enterprise world. This is my best attempt to debunk them.

Myth: Open Source is less secure because anyone can make changes

Although there are open source projects where anyone can freely make changes this is definitely not the norm.

Open source projects can be very selective with the contributions they accept.

Don’t believe me? Read this and this!

Myth: Open Source is less secure because anyone can see the code

Having your code open sourced, widely reviewed, and used by the entire world, won’t magically protect you from exploits. Even more, exploits also affect proprietary software where the code is not available.

That being said, having the ability of inspecting a library’s source before using it is way more secure than trusting a proprietary software vendor sales team.

Want to learn more? Read this, this and this.

Myth: Open Source is not production ready

This has been the hardest myth to debunk because of its absurdity.

Seventy-eight percent of respondents to the Ninth Annual Future of Open Source Survey of 2015 determined that their companies run part or all of its operations on open source software. It doesn’t get more production ready than that.

Myth: Open Source tools development stalls

If anything open source drives innovation even faster.

Example 1: Containers

Containers have proven to be the de facto standard for deployments. The open source community has been running containers for years. However, Windows based enterprises have had to wait several years for Microsoft to provide container support.

Example 2: Your CMS in the Cloud

Many enterprises are migrating their platforms to the cloud. Umbraco users can seamlessly host their existing websites on Azure with very little effort. WordPress has had PaaS support for ages. I wish the same could be said of other proprietary CMS.

Myth: Open Source is not a viable business model

These are just a few out of many successful companies that have been built on open source software:

Myth: Open Source is free

While downloading binaries for the majority of open source software is free. Releasing your product to production may not. The companies behind open source projects make their money with Corporate Support, VIP Hosting, or Commercial Subscriptions.

Myth: Open Source is hard for IT

This is debatable. As with anything different there will be a learning curve. The same can be said of proprietary tools with bad documentation and no access to the code.

Closing Remarks

Should you drop your proprietary stack and build everything with open source technologies? Certainly not. Unless a full rewrite is imminent and open source technologies can help you achieve your goals in a cleaner and faster way.
Should you favor open source over proprietary? It depends on the scenario. Always look at the problem you’re trying to solve and the impact of the tools on your organization.
Please stop mystifying open source tools as only good for hackers and startups. The open source ecosystem is more than prepared to solve your enterprise problems.

The Importance of Logs

Tue, 22 Mar 2016 00:00:00 +0000

Logs are one of the most important components of an application. They provide a quick and reliable way to verify behavior and diagnose errors. Properly written logs can be mined to take business decisions. And yet, they are often neglected.

Antipattern: No logs on a production running system

No matter what your software does, it should have logs. ¹ Logs will give you a quick way to verify the application behavior.

Antipattern: Logging ONLY errors

Logging errors is certainly necessary. However, normal system behavior should also be logged.

Why? Because it gives visibility of how the software is being used. It creates the infrastructure to monitor the business performance.

Example: This website serves approximately 15,000 registered users a day. Logging every Successful Login event gives the ability to configure an alert when the number of logins falls below 10,000. A low number of logins can impact the business and needs to be thoroughly analyzed. Moreover, if the number of Successful Login events goes above 25,000 there could be an issue with another part of the system such as the Session Storage Server.

Myth: Logging will impact performance

Garbage Collection too and that hasn’t stopped us from using it.

Although Logging can definitely add some minimal overhead not knowing how your system is behaving can certainly impact performance. ²

Best Practice: Auditable logs

Properly written logs clearly describe the chronological history of every user interaction since they started using the application until they finished.

Include in every log message enough information to quickly filter all the related events. e.g. In a web store this would be the sessionId. In a mobile banking app this could be a unique UUID generated when the application started.

Try to include as much information as possible to determine the log line originator such as the class and method.

Best Practice: Log Levels

Not every log line is written for the same audience. A business analyst could make little sense out of a stack trace. Some events are written to help engineers diagnose an intermittent issue while others are written to build business performance dashboards.

Segregate your log lines into their appropriate levels.

Level Selection Guidelines

Although the number of log levels may vary these four are almost ubiquitous to every development platform.

Error

The application encountered an irrecoverable error.

There should be alerts configured to notify the required personnel when any of these appear. An Error event should be treated as an outage. e.g. [ERROR] Database Connection Timeout

When logging errors try to include as much information as possible such as stack trace and method parameters. These will help you diagnose issues faster.

Warning

The application encountered a non-fatal error.

These should be monitored but are not urgent. e.g. [WARNING] 404 Not Found errors.

Alerts can certainly be configured on warning messages. For example an abnormal number of [WARNING] Failed Captcha Attempt could indicate a problem with the website rather than user error.

The number of Errors in your logs should be zero. Any abnormal behavior that is not an emergency should be a Warning. This practice will simplify your decision process.

Debug

Information to be read only by developers while diagnosing an issue.

e.g. [DEBUG] Model Info retrieved from the Database. This log level will help verify the behavior of the program but may add a lot of noise for non-debugging operations.

Info

Important application behavior.

This level would be read by developers as well as business analysts and log mining tools. This is the most important level because it provides the infrastructure to analyze the business performance in real time. e.g. [INFO] Payment Completed.

Keep you Info logs as readable as possible in a format that anyone can understand.

Best Practice: Log Aggregation tool

Gone are the days of manually searching through log files.

Several powerful log aggregation tools are available that can streamline your logs processing workflow. They take care of shipping the logs to a central location to be aggregated and searched through a nice web GUI. These tools come with lots of capabilities such as dashboards and alerts.

My personal favorite is Splunk. If its cost seems prohibitive, there are free alternatives such as the ELK Stack.

Best Practice: Log key value pairs as opposed to events

A lot of benefit can be reaped from logging events as key value pairs. Favor the following:

[INFO] Actor=LoginService; Action=Login_Attempt; Result=True
[INFO] Actor=LoginService; Action=Login_Attempt; Result=False

in opposition to:

[INFO] LoginService; Login Succeeded
[INFO] LoginService; Login Failed

Key value pairs can greatly help you aggregate and analyze events in a similar way to a database table.

Summary

Properly written logs are a powerful tool that can generate realtime feedback for the business as well as for the technical team. Use them wisely to unleash their full potential.

Sometimes, it is impractical for applications to write anything due to their low latency and high volume requirements. Chances are your app is not one of those. Regardless, every application needs a way to verify its behavior. ↩
Nine out of ten times your production systems will have more than enough capacity to handle the load so you won’t have to tweak anything. If performance becomes an issue because of the logs buy more/better hardware. Just leave the logs alone. ↩

General POS design considerations

Tue, 02 Feb 2016 00:00:00 +0000

For the first years of my career I built a few Point of Sale (POS) systems using C# and WPF. All of these systems had the same working principle: the software was installed in a remote terminal that customers would use to make transactions. They also shared the same pain points such as updates, database migrations, and physical devices reliability. The following are lessons learned from painstaking experiences and a few nights without sleep. The earlier in the project these get incorporated the more stable the overall system will be.

TL;DR: Building a POS system is hard. Brace yourself.

Support

Distributed systems are very hard to manage. [citation_needed] The better your support tools are the more stable your system will be. ¹ Invest time building support tools rather than fire-fighting outages.

Comprehensive and detailed transactional logging

Logs are an important part of every POS system. They are the backbone of the verification and diagnostics process. Unfortunately, not all logs are properly written nor actively read. Gone are the days where logs were only used to debug errors. Integrate a log aggregation tool such as Elastic Search or Splunk into your overall design.

For traceability and debugging purposes, it is important to correlate related events. Devise a way to link all the events related to the same transaction, whether using the same Id field in multiple log lines or logging everything all at once in the same log line. This practice will help you automate system performance profiling.

Last but not least, avoid having multiple log files. They are harder to read.

System heartbeat

Make each terminal periodically transmit a heartbeat with all the relevant diagnostics information. This heartbeat will allow you to create a comprehensive dashboard of the entire ecosystem health.

Moreover, you’d be able to proactively fix errors such as shipping supplies when the printer is low on paper.

Profile memory consumption early and frequently

Almost every system running for long periods of time will be subject to some sort of performance degradation due to memory leaks.

Readily available remote access to the terminals

Every once in a while a support technician needs to log into a terminal to diagnose a weird issue or help a customer setup something. The frequency of these interactions is inversely proportional to the system robustness.

Plan for a mechanism to remote into the terminals and control their screen. TeamViewer has a very comprehensive product that can maintain a database of terminal connection details.

Lock and control every software aspect

Customers can be your worst enemies when fiddling with the system. Lock everything in the remote machines. Disable the keyboard. Disable the USB ports. It will save a lot of support calls.

If possible ship a clean Ghost image embedded into the machine, so that troubleshooting becomes as simple as re-image the computer and apply the configuration.

One-Click Diagnostics tool

Build a One-Click Diagnostics tool that users and support personnel can use to verify all the components are working as expected. Include new dependencies in the tool before adding them to production.

Reprint anything

Design the system in a way that can reprint any receipt as the customer saw it. Build some functionality to make these receipts available from a website.

Rotate your developers through support

Make your developers take support calls. Incorporate their feedback into subsequent updates. Your system will become more robust and resilient in a blink of an eye. ²

Updates

Updates are the backbone of distributed systems. Upcoming system versions should make it to the remote locations in a fast and reliable manner.

Embedded updates system from day one

Consider the updates system from the early project stages. Integrate this system into the main application. Don’t build it as a black box. Beware, this is not a permission to build a monolithic application.

Design the updates system in a way that doesn’t interrupt the business flow. If possible implement online updates that require none to very little downtime.

Many locations have their own IT Security team with all sort of specific firewall rules. Always download files over HTTPS on port 443 to streamline terminal setup.

Limit the number of external dependencies. Upgrade them as regular application updates.

Limit the number of external dependencies as much as possible. External dependencies impose a big burden on system and compatibility testing. Automate their installation as part of the update process. It will prevent a lot of support calls.

One-click installation/configuration

This cannot be stressed enough. It will prevent countless user input errors. It is one of those things that once you have it cannot be lived without.

Same binaries for updates and installation

This will simplify your build, deployment, and support process. Make the installer smart enough to detect when a brand new installation or an upgrade is taking place.

Databases

There will come a time when the terminals will need a local database. Try to limit their usage to store the data that hasn’t been transmitted. Databases come with their fair share of issues too.

Use an embedded lightweight database

Avoid heavyweight systems such as SQL server, it will eat up your memory. Moreover, it will be a pain to upgrade. Choose an embedded database system such as SQLite which is easier to maintain.

Easy db migrations

Your schema will change. Be prepared.

Configuration

Configuration problems can bring any system to halt. Manage your configurations with an iron fist.

Exactly one readonly configuration that can only be changed by developers

The readonly part is very important, it would guarantee the integrity of the system once it gets deployed. The last thing you want is customers fiddling with settings such as COM port speed or toggling ON unfinished features.

Avoid having multiple configurations it will make troubleshooting and support harder.

Exactly one read/write configuration that can change at any time

This would be the configuration for things your users can change such as the printer COM port or the terminal number that identifies the store.

Design your system in a way that these configuration changes can occur on the fly without the need of a restart or any other provision. This will save countless hours of troubleshooting. Moreover these configuration changes can potentially be remotely triggered by support personnel.

The configuration for each individual location should be transmitted and accounted for in the servers

Uncountable benefits can be achieved by storing all the configurations in a centralized location.

On the event of hardware failure the configuration could be restored into a separate machine and the business could be up and running in minutes. Configuration errors could be pro-actively diagnosed by scheduled jobs.

Development

The following are development guidelines to keep in mind. In retrospective they look a lot like Continuous Integration best practices.

Dev should not use absolute paths

Never use absolute paths in the code. It will prevent you from having multiple versions of the system installed in the same box. It will lock you down to a specific folder structure. Enforce this practice with strict code review.

Version number should be specified in only one place

Have a single version number in a single place. The database and the software should have the same version. Display and transmit this version number as frequently as possible.

Automated builds

I remember working in a system where Build Day was Pizza Night. Please don’t do that. Continuous Integration - Automate the Build.

Automated releases

Continuous Integration - Automate Deployment

Move functionality from the terminals off to the servers

Terminals should only process transactions. Reports and other sorts of housekeeping functionality should be implemented on the servers. This will allow the system in the store to be operational while other housekeeping tasks are in progress. Consequently, system enhancements and updates will be less costly because it is easier to update a few servers than ship binaries to thousands of stores.

Binaries signing

Sign your binaries during the build process. It will prevent malicious users from tampering with them and compromising your system.

Compiler Warning CS0659

Sat, 30 Jan 2016 00:00:00 +0000

The majority of C# developers have found Compiler Warning CS0659 at some point of their careers. Ignoring this warning can produce unexpected program behavior. The following is summary of the impact this warning can have on a codebase and how to prevent it.

TL;DR: Override GetHashCode whenever Equals is overridden

Reference Equality

Let’s consider the following Dog class in C#.

public class Dog
{
    public int Weight { get; set; }
    public string Name { get; set; }
}

And the following usage of it.

var fido1 = new Dog { Name = "Fido", Weight = 12 };
var fido2 = new Dog { Name = "Fido", Weight = 12 };
Console.WriteLine(fido1.Equals(fido2));

The snippet of code from above prints False. For reference types the default implementation of Equals compares the references instead of the values where they are pointing to. ¹

Github Commit 5d488e6d6127111167e9a8f36fe83a771e36cb9c contains the sample source code up to this point

Semantic Equality

Let’s enhance the Dog class so that comparisons between Dogs have a better semantic meaning.

public class Dog
{
    public int Weight { get; set; }
    public string Name { get; set; }
    
    public override bool Equals(object obj)
    {
        var that = (Dog)obj;
        return this.Weight == that.Weight && this.Name == that.Name;
    }
}

After overriding the Equals method the following code prints True.

var fido1 = new Dog { Name = "Fido", Weight = 12 };
var fido2 = new Dog { Name = "Fido", Weight = 12 };
Console.WriteLine(fido1.Equals(fido2));

Github Commit 7abeb0d9b8dbf7c09124485fb0f7e3bd18257b18 contains the sample source code up to this point

Warning CS0659

Once we compile the code we get the following warning.

warning CS0659: ‘Dog’ overrides Object.Equals(object o) but does not override Object.GetHashCode()

Compiler Warning (level 3) CS0659 is straightforward and direct to the point: If you override Equals you need to override GetHashCode.

The Problem

This is not one of those warnings that can safely be ignored. ² It will come back to bite you. Let’s consider the following usages of the Dog class.

var dogShelter = new Dictionary<Dog, int>{
    {new Dog { Name = "Fido", Weight = 12 }, 100},
    {new Dog { Name = "Pete", Weight = 5 }, 10}
};
var fido = new Dog { Name = "Fido", Weight = 12 };
Console.WriteLine(dogShelter.ContainsKey(fido));

The code from above prints False because fido cannot be found in dogShelter.

Why?

Overriding Equals is not enough. The GetHashCode method provides a numeric value for quick equality checks such as those used in collections like Dictionary and HashTable. ³

Github Commit d91504bfedcb60ab86fffe72dcac88fb775405de contains the sample source code up to this point

Solution

Override GetHashCode whenever Equals is overridden. The following is an implementation of Dog where GetHashCode is overridden. ⁴

public class Dog
{
    public int Weight { get; set; }
    public string Name { get; set; }
    
    public override bool Equals(object obj)
    {
        var that = (Dog)obj;
        return this.Weight == that.Weight && this.Name == that.Name;
    }
    
    public override int GetHashCode()
    {
        unchecked
        {
            return (Weight * 397) ^ (Name != null ? Name.GetHashCode() : 0);
        }
    }
}

With this implementation of Dog in place, fido can be found in the dogShelter.

var dogShelter = new Dictionary<Dog, int>{
    {new Dog { Name = "Fido", Weight = 12 }, 100},
    {new Dog { Name = "Pete", Weight = 5 }, 10}
};
var fido = new Dog { Name = "Fido", Weight = 12 };
Console.WriteLine(dogShelter.ContainsKey(fido));

Github Commit 98c9adc1fff2aee2362fd494f3a43cdb7f4d3d8a contains the sample source code up to this point

In a nutshell

Object.GetHashCode is as important as Object.Equals for equality matters. Never override one without the other.

Update: Equality is Hard ⁵

Since Dog is a mutable object GetHashCode will return different values when either of its properties change making it impossible to be found in dictionaries. Down the rabbit hole we go. To overcome this issue we have to make its properties readonly.

public class Dog
{
    public int Weight { get; }
    public string Name { get; }

    public Dog(string name, int weight)
    {
        this.Name = name;
        this.Weight = weight;
    }

    public override bool Equals(object obj)
    {
        var that = (Dog)obj;
        return this.Weight == that.Weight && this.Name == that.Name;
    }

    public override int GetHashCode()
    {
        unchecked
        {
            return (Weight * 397) ^ (Name != null ? Name.GetHashCode() : 0);
        }
    }
}

Now our Program looks as follows:

var dogShelter = new Dictionary<Dog, int>{
    {new Dog("Fido", 12), 100},
    {new Dog("Pete", 5), 10}
};
var fido = new Dog("Fido", 12);

Console.WriteLine(dogShelter.ContainsKey(fido));

Github Commit dffbd97927070b05c623192a788a33b97e7101c7 contains the sample source code up to this point

Value Types to the Rescue

An easier way to get rid of all of this Equality nonsense would be to make Dog a value type. That way we’d get equality for free.

public struct Dog
{
    public int Weight { get; }
    public string Name { get; }

    public Dog(string name, int weight)
    {
        this.Name = name;
        this.Weight = weight;
    }
}

Github Commit b756e87a63abec9ea8c069bc5d9c09b2a2376311 contains the sample source code up to this point

Object.Equals Method (Object) ↩
Warning levels, what are they, which level to use, and should they be treated as errors? ↩
Object.GetHashCode Method () ↩
ReSharper users can benefit from automatic generation of GetHashCode implementations ↩
The Right Way to do Equality in C# ↩

Waste Prevention

Fri, 29 Jan 2016 00:00:00 +0000

Waste is the enemy of value. It is usually manifested as building the wrong product. The following are two waste prevention techniques I’ve learned during my years as an engineer.

This is the third article of the Deliver Value Fallacy series. Part 1 covers Delivering Waste and Part 2 covers Technical Debt.

Data Driven

Aim to back every decision with data. While not always possible, most of the times it is possible. Engineers are as responsible for the company as everybody else. Always -politely- question business decisions made out of sheer inspiration. Be ready to educate and share how data driven decisions help the business while intuition can be wrong and costly.

Simplify. Simplify. Simplify

Engineers have a tendency to overthink problems. Ask business owners questions about their decision drivers. Learn the business needs. With the clarity these provide you can simplify design aspects that are not really needed.

Examples

The following are real world scenarios I’ve been exposed to recently. ¹ They illustrate how the waste prevention measures can be applied to achieve optimal results that help the business move faster in the correct direction.

Example 1: The CMS

The business: Rebuild the website using the company’s CMS.
Engineers: Why do you need a CMS? CMS’es are hard to test, their upgrades are very costly, they impose a great burden on the infrastructure, and they’re very hard to move away from.
The business: We need a CMS because we need to frequently update the images of the website. And you cannot do it because you’re always busy.
Engineers: How often do you have to change those images? Is there anything else that you frequently need to change other than the images?
The business: Just the images on a quarterly basis.
Engineers: What if we block our calendar one week per quarter so we can work together updating the images?
The business: Unacceptable ಠ_ಠ.
Engineers: Would you be willing to look into solutions other than a CMS that would allow you to change the images in a day? Granted, they would not be as easy to use as your very expensive CMS but they would certainly take less time to build. Plus we’d properly train you on how to use it.
The business: Maybe, but we need to make sure it does all that we want.
Engineers: Fair enough, we’ll schedule a meeting in three days to showcase a potential solution.
The business: Mutters, ineligible.

Example 2: Database all the things

The business: We need to start paying attention to the customer’s date of birth (DOB). Let’s save it in the database and create some Crystal Reports off from it.
Engineers: Whoa! Hold your horses. A database field? What is it for? Why do you need this new information?
The business: We want to be able to create reports of transactions linked to the customers birthday. We want to study if there’s a correlation between birthdays and sales. If it turns out that people buy themselves expensive gifts before their birthdays we want to kick off a new marketing project to exploit this trend.
Engineers: Excellent idea. However, to be able to create Crystal Reports off customer’s DOB we’d need to write code to save it in the database. Then we’d need to modify the database schema across all the different database environments. We’d need to create a change request with the DBA team for this. Once the data starts flowing we’d need to involve the Reporting Team and bring them up to speed with the database schema and the relationships between entities. Based on the weather conditions this could take from four to six weeks.
The business: Obviously annoyed slowly turns red…
Engineers: There’s an alternative though. It would help us get feedback faster. What if we log the customers DOB and we create the reports using our current installation of Elastic Search (ELK). All this would take from us is writing a log line. Once the data starts flowing we can sit together to create the ELK reports. You’d have enough data for the marketing initiative in two weeks at most.
The business: But these ELK reports don’t feel real. They’re not coming from a database.
Engineers: ELK is a database too. It is simply a log-based database. ELK and other logging aggregation tools can be a compliment to traditional reporting and database tools. Important business decisions can be made based on ELK data as well as on SQL data. The business: When I was a developer we used to store everything in SQL.
Engineers: How about the following approach: Let’s implement the ELK reports first. If after two weeks you don’t have all the necessary information to kick off your marketing campaign, we’ll store the DOB in the database.
The business: Mutters, ineligible.

Example 3: Architectural decisions

Happy Trigger: Let’s have an architectural discussion to make sure that all our web apps can be loaded as angular modules.
Senior Engineer: There’s something I don’t understand. You want to create a single web app that would be the entry point to all of the other web apps?
Happy Trigger: Exactly!
Senior Engineer: What for?
Happy Trigger: To reduce code duplication. And reuse the same CSS resources.
Senior Engineer: Interesting. Wouldn’t that create a monolithic application with a single point of failure?
Happy Trigger: Not really because each module can…
Senior Engineer: Interrupts hurriedly We have over thirty different apps. Some of them were built way before angular ever existed. They serve very different business functions there’s very little code duplication going on. I’m sure there are other ways of sharing CSS resources that don’t involve angular. Have you looked into those?
Happy Trigger: Angular is cool. We can use npm and bower and make web APIs for everything.
Senior Engineer: What business problem does this Unification Initiative solves?
Happy Trigger: It would create an architectural framework.
Senior Engineer: Wouldn’t locking every application into a single technology be a bad idea? I heard there’s a new completely incompatible version of angular around the corner. We need hard numbers to justify this to the business. It needs to pay off technical debt or create business value. An architectural framework helps nobody unless it is really needed.
Happy Trigger: Mutters, ineligible.

Foreword

That’s it, no magic bullet. Data Driven Decisions and Simplification are simple concepts that will save countless hours and headaches. Do you have to always push back on everything? For God’s sake no. However, keep an open eye for situations that can create waste.

The details are obfuscated to protect the innocent. ↩

The Deliver Value Fallacy: Technical Debt (Part 2)

Thu, 28 Jan 2016 00:00:00 +0000

The term Deliver Value is widely [ab]used. It is regularly used as an excuse to avoid following engineering best practices. In this second installment I plan to illustrate some of the common misconceptions associated with Delivering Value and Technical Debt.

Please head to the first post to learn about Delivering Waste.

Debt

something, typically money, that is owed or due.

A tale of two companies

YouSail, Inc. and GreatShoes, LLC. are two different companies in the same position: Online sales are low. ¹

In a nutshell YouSail, Inc. is big on outsourcing but relies heavily on business metrics while GreatShoes, LLC. has a strong engineering culture that gets usually overruled by intuition and lack of data. Both of the companies devise relatively different plans to address their website issues.

The following is a summary of the website redesign roadmap each company plans to implement. Full details behind the decision processes and company cultures can be found in the Part 1 of this series.

Company tally

	YouSail, Inc.	GreatShoes, LLC.
Reaction Time	Six Months	One Year
Decision Time	One Week	Multiple Weeks
Root Cause Analysis	Data Driven. Payment Flow	No Data. Entire Design
Time to Feedback	After Every Single Page	After project completion
Projected Completion	Six Months	Three months

Delivering Value

Which of the two teams is delivering more value to the business?

The cultural differences between the two companies clearly impact their performance. The well-qualified team behind GreatShoes, LLC. has built a highly-maintainable product. The initial engineering plan would rebuild the entire site -on top of the existing one- in the same time that it would take YouSail, Inc. to get a single portion of their website rebuilt. The good engineering principles applied over the years empower the business to quickly adapt and react. Moreover, this is a professional team that understands risks and proposes an adequate plan to minimize them.

Technical Debt is simply that: Debt. It needs to be paid off or will eventually bring businesses down to a halt. If the executive team behind YouSail, Inc. would regularly spend more time and energy on their website they’d be able to replace the entire Payment Flow in under a month.

Sometimes it can be very hard to condone the actions behind YouSail, Inc.. Yes, they’re a making decisions based on business metrics and data but the state of neglect in which they keep their website is significantly hurting their bottomline. ²

Delivering value does not mean cutting corners. On the contrary, Delivering Value is about delivering just enough functionality just in time to meet the business needs, while maintaining high engineering standards that will allow the product to adapt as needed.

Visit Part 3 of this series to learn how to Prevent Waste.

Not their real names. These scenarios are based on real world companies but the details obfuscated to protect the innocent. ↩
The Phoenix Project illustrates the struggles of a similar organization with financial problems. It is a great read that clearly paints the need to involve and consider IT in the decision process. ↩

The Deliver Value Fallacy: Waste (Part 1)

Tue, 26 Jan 2016 00:00:00 +0000

The term Deliver Value is widely [ab]used. It can be a wildcard to avoid following engineering best practices or build utterly complex solutions that solve no real business problem. In this multi-part article I plan to illustrate some of the common misconceptions associated with Delivering Value. The series will start with delivering waste.

Waste

an act or instance of using or expending something carelessly, extravagantly, or to no purpose.

A tale of two companies

YouSail, Inc. and GreatShoes, LLC. are two different companies in the same position: Online sales are low. This is how they plan to solve their problem. ¹

YouSail, Inc.

YouSail, Inc. rents luxury yachts. The executive team reasons that since they’re not a technology company there’s not a clear need for a technology team. ² They strongly believe in paying specialized vendors for their expertise rather than having to deal with more employees. Their online sales have been down for the last six months.

The Website: The company’s website has always been outsourced. Countless contracting companies have worked on it over the years. It is a tree branch waiting to fall made out of stitched pieces. The fact that it works is a great indicator of how good technology has become over the last years. In spite of not owning the website development, YouSail, Inc. has a strong hold over important business metrics, backed up by lots of analytics.

The redesign: A quick analysis of the metrics indicate that traffic is up. More people are starting reservations. Nevertheless, they’re abandoning the rental during the payment flow. A plan to redesign just the payment section of the website is kicked off in under a week. The redesign project will work on a single page at a time. Moreover, the two versions of the page will be simultaneously served to validate that the new website is working better than the old one. All of this development work will be contracted out to a development agency from Argentina. The projected completion date is six months.

GreatShoes, LLC.

GreatShoes, LLC. sells shoes online. They pride themselves in being agile and Deliver Value frequently. The technology department is the second largest department in the company with dozens of well-qualified highly-trained technology professionals. The CEO can often be heard saying: “nowadays, every company is a technology company”. And he’s right to say so because most of the company’s income comes through sales made directly to customers from their website. Unfortunately, sales have been on a steady decline for over a year.

The redesign: The business owners think sales are down because the website does not look chic enough. Finally, after several weeks of approval meetings and budget lobbying they kick off a plan to redesign the entire website. Besides, they think that they need a CMS. How is a CMS going to help with the sales? They reason that by changing the content frequently customers should be more engaged. They ask their development team to come up with a plan.

The proposal: Fortunately, the development team at GreatShoes, LLC. understands the risk behind a big redesign like this. They propose to gradually change each individual portion of the site while comparing its performance against the old version. Only when the redesign is completed they will start to incorporate the CMS one page at a time. The engineers thought this proposal would have the least risk and would gather feedback from customers faster. The projected completion date is six months as well.

Request Denied: Business owners think the whole incremental approach would add a lot of overhead. Plus they want their CMS from day one. They don’t understand why a CMS is harder to test nor to maintain in the long run. The proposal is unacceptable. They graciously ask the development team to build everything at once in three months.

Company tally

	YouSail, Inc.	GreatShoes, LLC.
Reaction Time	Six Months	One Year
Decision Time	One Week	Multiple Weeks
Root Cause Analysis	Data Driven. Payment Flow	No Data. Entire Design
Time to Feedback	After Every Single Page	After project completion
Projected Completion	Six Months	Three months

Notice how the projected completion date is closer for GreatShoes, LLC. -half of YouSail, Inc.’s-. They are going to have an entire new website in three months. And yet it is all going to waste because they don’t know that they’re losing sales because of the website, or because they need a mobile app, or because the competition has better prices. They simply don’t know. Moreover, there are no clear plans to incorporate performance feedback loops that can help steer the project based on customer needs.

YouSail, Inc. is happy to spend six months in rebuilding just a single portion of their entire site. The cost is hefty but they know that every step they make in that direction will be validated with an increase in sales. Moreover, the risk is low. If the project cost gets too high or the business needs to focus on another direction, they can still benefit from whatever new pages they got time to build.

Delivering Value

Which of the two teams is delivering more value to the business?

After a simple analysis it becomes clear that the team from Argentina is delivering more value to the business. Simply because they are building the correct product. They are solving a clear and measurable business problem. On the other hand the second team is building waste. They are making decisions based on intuition as opposed to measurable metrics.

Visit Part 2 of this series to learn Delivering Value and Technical Debt.

Not their real names. These scenarios are based on real world companies but the details obfuscated to protect the innocent. ↩
The Phoenix Project illustrates the struggles of a similar organization with financial problems. It is a great read that clearly paints the need to involve and consider IT in the decision process. ↩

The best book I read in 2015

Fri, 22 Jan 2016 00:00:00 +0000

The Lean Startup -by Eric Ries- diametrically changed many of the notions I had about how software projects should be carried on.

It helped me understand what being agile truly means. True agility means going full cycle -from idea to the customers- in two weeks or a month instead of the infamous timeline nonsense. Agile teams control their deployment and technology stack.

The Lean Startup movement is based on the importance of reducing waste during the product development lifecycle. The book showcases how building the wrong product is the ultimate form of waste. The only effective measure against waste is ship often and validate with the users at every step.

The book depicts with multiple examples from the real world how product ideas are initially wrong in most of the cases. Preconceived notions of how customers will use the products are generally wrong. Yes, your idea is probably wrong too! Moreover, businesses should not get too attached to an idea, once the growth stops, businesses should be ready to pivot into a different direction.

Be not confused with its title. The teachings of this masterpiece can also be applied in large and well established enterprises. It is certainly a mandatory read for every technology professional.

Tdd Apps

Book Review: Cracking the Coding Interview

Exercises

Big-O notation

Be prepared

Lessons from building a Raspberry Pi 4 NAS

USB

File System

Operating System

Power Supply

Automation

Learn

South Florida's Codecamp 2020

Presentation Outline

Solarize Slack for Real

Before

After

Step 1: Change the Sidebar theme

Step 2: Change the Background Colors

Ubuntu Server 18.04 LTS upgrade issues and their solutions

Upgrade Process

Problems during the upgrade

/boot partition is full

Problems after the upgrade

Local DNS configuration disappeared

KVM virtual machines failed to start

Laptop screen does not turn off after long inactivity

Deprecated PostgreSQL package

Bottomline

Migrating large files to GitHub

Prerequisites

Solution

GH001 Error

Success!

Got other errors?

Want to know more?

Serverless API Part 1: From idea to Production

The Ivory Tower Architect

Why so complicated?

The MVP Architecture

BONUS: It is always the Database

Summary

Book Review: TDD by Example

The Author

Part I and II: Examples

Stress

Part III: Patterns

Rapid Feedback

Refactoring

Mastering TDD

Should you read it?

Book Review: Clean Architecture

The Author

The Stories

Architecture

Details

Should you read it?

Book Review: Rich Dad Poor Dad

Good

Bad

The Heartbeat Pattern

Which computers in a global network are offline?

How do scientists know black holes exist?

Tracking transactions

Heart beats

Heartbeat Pattern

Advantages

Reference Architecture

Client

API

Alerts Recurrent Task

Book Review: Creative Selection

A software engineer

Beware of doing things the Apple way

Collaboration and Secrecy

Team size

Planning

Convergence

Patents

Conclusion

`/boot` partition is full

`GH001` Error

`DebugEFCore` internals

`LazyPropertyHelper` nuget to the rescue