View this email in your browser

Software Weekly - 11/05/2018

Human Scalability, Uber's data platform, serverless backend checkout & more


Last week’s most popular episode: Diffbot: Knowledge Graph API with Mike Tung. Mike talks about building AI applications and how he came to build Diffbot. Diffbot crawls the entire web and uses Natural Language Processing and Understanding to assess the information. This great episode shows how far we have come in AI.

Old favorite: WebAssembly with Brendan Eich. Brendan, the creator of JavaScript, talks about WebAssembly, a way to compile and run any code in the browser. Brendan also talks about Brave, a faster and safer browser.

Datadog, a monitoring and analytics platform that integrates with more than 250 technologies, correlates metrics, distributed traces and logs into one platform. Datadog makes it easy for teams to monitor every layer of their stack in one place, but don't take our word for it—start a free 14-day trial today.

Read: From Software Engineering Daily

Serverless Instant Checkout Links with Square by Richard Moot from Square. Explore how Square built an entire Instant Checkout experience using GCP Functions and Firebase. Complete code examples provided!

Serverless Backend using AWS Lambda: Hands-on Guide by Rajat S. Step-by-step guide on launching an AWS Lambda + DynamoDB with detailed code examples and intro serverless framework.

Powering Search and Discovery at VICE by Michael Henderson from Vice Tech. Explore how VICE Media uses Elasticsearch to power their search, augmented with Kinesis Streams and Lambda for ingestion. Interesting look at the architecture and considerations for a search service.

Uber’s Big Data Platform: 100+ Petabytes with Minute Latency by Reza Shiftehfar from Uber. Go through different architectural generations of Uber's big data platform. Uber rearchitected multiple times to scale better and get very low latency. They now use Kafka, Hudi, as well as Hive, Spark, and Presto.

Building a Big Data Pipeline With Airflow, Spark and Zeppelin by Mahdi Karabiben. Mahdi takes us through building data pipelines in detail, explaining components and wiring up Airflow, Spark, and Zeppelin.

Stay tuned for more episode summaries, digests, longform articles, and other great content from the SE Daily blog. What else do you want to read about? Let us know.

Attend DataEngConf ‘18 the No-Bullsh*t Data Event designed for and by data engineers and scientists. Taking place this year at Columbia University in NYC on Nov 8-9th. With 4 dedicated tracks: Data Engineering, Data Science, AI Products and the brand new Hero Engineering. Join geeks from the west & east coast and receive a $200 discount using code "SWEngWeekly" at checkout.


What's New in Software Engineering

GitHub posted their October 21 post-incident analysis. GitHub experienced an outage for over 24 hours, caused by a brief 43-second connectivity outage that had cascading effects. "Our strategy was to prioritize data integrity over site usability and time to recovery." The network partition broke MySQL replication topology, degrading performance and ultimately taking hours to restore from backup. GitHub is now going active/active.

React introduces Hooks. Although still a proposal with an implementation only in an alpha branch, developers are going crazy online for React Hooks. Everyone is super excited about how Hooks reduces component complexity. Many developers started building Hooks integrations for their libraries. Warning: Don't take a dependency on Hooks in production. They are not ready yet! 

Google launches AdaNet for combining algorithms with AutoML. AdaNet is an open source tool that provides a general framework for obtaining better AI models, using ensemble learning to combine and improve machine learning models. Check it out on GitHub.

React Native publishes new Open Source Roadmap. The plan is to get React Native to be more React and less Facebook.

Amazon Rekognition Announces More Accurate Object and Scene Detection. Some awesome additions to Rekognition best demonstrated by Jeff Bar, with more accurate detection (and bounding boxes for each object), count occurrence of an object and hierarchical relationships

Interesting Picks in Software Engineering and Architecture

Reinforcement Learning with Prediction-Based Rewards by OpenAI. AI agent learns to play Mario without rewards. It was incentivized to avoid boredom by curiously exploring the environment. It defeated the boss. My recommended read this week.

Designing Features for Mature Systems: Lessons Learned from Manta by Jordan Hendricks. Great video on how Manta (Object Storage by Joyent, similar to S3) is built. Manta has both storage and compute. A simple explanation of a complex system.

Parsing logs 230x faster with Rust by André Arko. André writes a log parser in Rust to extract data, deploys it to Lambda, and runs everything for free. Interesting read as progression from using Spark to Lambda.

Building the Contacts Platform at LinkedIn. LinkedIn takes us through how to build a contacts platform with a lot of focus on how they built the data model and chose to lay out the data. They also mention how they migrated the data to the new system. Terrific work.

Testing Privacy-Preserving Telemetry with Prio at Mozilla by Robert Helmer. Learn how Firefox browsers publish analytics to the Prio system in the public domainaggregating data before sending it to Mozilla backend servers. Mozilla delivers on the promise of privacy, but I am a little bit worried that their engineering blog is called Hacks...

Scenes from distributed systems. Click the link for a laugh at this comic by Julia Evans. Fun Post!

Start Performance Budgeting by Addy Osmani. Similar to SRE Error Budgeting, this performance budgeting philosophy will make sure you run an application that both developers and users love.

Get ready to build content-rich websites and professional web applications with Wix Code. Store and manage unlimited data with built-in databases, create dynamic pages, make custom forms and take full control of your site’s functionality with Wix Code APIs and JavaScript. Plus, now you can get 10-percent off your Premium plan. Go to

What's New in Tech

HashiCorp raises $100 at $1.9 billion valuation in a focused push on multi-cloud infrastructure automation. Though Terraform is their most popular product, Hashicorp provides Consul, Nomad, and Vault to help connect, run,and secure applications.

Google walkout: Employees protest over sexual harassment scandals. Employees across the globe walked out in a protest demanding better accountability and more representation.

China State-Owned Company Charged With Micron Secrets Theft. A Chinese state-owned company was charged with stealing trade secrets from US Micro Technology. Chinese economic espionage against the US is growing.

Someone posted private messages from 81,000 Facebook accounts. Someone posted details for 120 million Facebook accounts for 10 cents each and found more than 81,000 included private Facebook messages.

OpenShift is a Kubernetes platform from Red Hat. OpenShift takes the Kubernetes container orchestration system and adds features that let you build software more quickly. OpenShift includes service discovery, CI/CD, built-in monitoring and health management, and scalability. With OpenShift, you avoid getting locked into any particular cloud provider. Check out OpenShift from RedHat, by going

Hi there, I'm Abdallah and I a new editor @software_daily. I am a Software Developer myself and love talking news, insights, and analysis of software engineering. We are expanding the Software Weekly newsletter to cover the more interesting content from across the web.

If you enjoy the new additions, got something interesting to share, or have comments or suggestions, hit the reply button & let us know. You can also contact Abdallah  ( or Jeff ( or @software_daily

Thank you for your support,
Abdallah Abu-Ghazaleh (@TheGhazStation)
Join the Slack community and discuss your thoughts with fellow listeners.

Newsletter Feedback

 How would you rate the recent changes to the Software Weekly Newsletter?
lowest 1   2   3   4   5   6   7   8   9   10   highest
Sorry, voting is closed.

If you enjoy the newsletter, please forward it to your friends.
What topics are you intersted in learning more about through the Software Weekly Newsletter?
 AI, Machine Learning
 Data Science 
 Containers, K8s
 Observability, Monitoring
 Chaos Engineering
 DevOps, SRE
 Other (reply to email with response)
Copyright © 2018 Software Daily, All rights reserved.

Want to change how you receive these emails?
You can update your preferences or unsubscribe from this list

Email Marketing Powered by Mailchimp