Ramblings of an indie open source business owner. https://www.rathboma.com Parquet provides significant benefits for sparse reads of large datasets, but is it always the file format to use? Fri, 20 Dec 2019 00:00:00 +0000 https://www.rathboma.com/2019/12/20/parquet-or-bust.html https://www.rathboma.com/2019/12/20/parquet-or-bust.html hadoop File formats can be confusing, so lets delve into Columnar file formats (like Parquet) and explain why they're different to regular formats (like CSV, JSON, or Avro) Thu, 21 Nov 2019 00:00:00 +0000 https://www.rathboma.com/2019/11/21/guide-to-columnar-file-formats.html https://www.rathboma.com/2019/11/21/guide-to-columnar-file-formats.html hadoop I'll talk through the basics of Threads, Akka, Futures, and Timers in this quick overview of concurrency for Scala. Great for those building apps in Scala. Tue, 28 Mar 2017 00:00:00 +0000 https://www.rathboma.com/2017/03/28/scala-concurrency-options.html https://www.rathboma.com/2017/03/28/scala-concurrency-options.html scala Scala's apply functions are commonly seen alongside case classes, but they can do so much more. Here are 4 fun ways they are used in Scala. Mon, 06 Mar 2017 00:00:00 +0000 https://www.rathboma.com/2017/03/06/scala-object-apply-functions.html https://www.rathboma.com/2017/03/06/scala-object-apply-functions.html scala While Scala is amazing it has an overwhelming number of features. These books and on-line resources will help you learn and perfect Scala whether you're coming from Java, Python, Ruby, or any other language. Tue, 14 Feb 2017 00:00:00 +0000 https://www.rathboma.com/2017/02/14/scala-books.html https://www.rathboma.com/2017/02/14/scala-books.html scala Apache Spark is a powerful technology with some fantastic books. I'll help you choose which book to buy with my guide to the top 10+ Spark books on the market. Fri, 13 Jan 2017 00:00:00 +0000 https://www.rathboma.com/2017/01/13/spark-books.html https://www.rathboma.com/2017/01/13/spark-books.html spark I'll walk through what we mean when we talk about 'storage formats' or 'file formats' for Hadoop and give you some initial advice on what format to use and how. Thu, 01 Sep 2016 00:00:00 +0000 https://www.rathboma.com/2016/09/01/a-beginners-guide-to-hadoop-storage-formats.html https://www.rathboma.com/2016/09/01/a-beginners-guide-to-hadoop-storage-formats.html hadoop Confused about whether Map Reduce is one word or two? Let me settle this once and for all. Wed, 29 Jun 2016 00:00:00 +0000 https://www.rathboma.com/2016/06/29/mapreduce-or-map-reduce.html https://www.rathboma.com/2016/06/29/mapreduce-or-map-reduce.html hadoop Scalding's scala-like API lets us join and summarize data quickly and easily, but beware -- it looks like regular scala, but sometimes behaves differently. Wed, 02 Mar 2016 00:00:00 +0000 https://www.rathboma.com/2016/03/02/Scalding-Typesafe-Tutorial.html https://www.rathboma.com/2016/03/02/Scalding-Typesafe-Tutorial.html hadoop Joining and analysing data in Hadoop using Python MapReduce. I compare this solution to the same solution in other MapReduce frameworks. Tue, 09 Feb 2016 00:00:00 +0000 https://www.rathboma.com/2016/02/09/python-tutorial.html https://www.rathboma.com/2016/02/09/python-tutorial.html hadoop Software engineers from Cloudera, Foursquare, Spredfast, JauntVR, and Elondina chat with me about their favorite Hadoop MapReduce frameworks and why they like them. Tue, 05 Jan 2016 00:00:00 +0000 https://www.rathboma.com/2016/01/05/experts-and-mapreduce.html https://www.rathboma.com/2016/01/05/experts-and-mapreduce.html hadoop To follow my post implementing a pipeline in regular Spark, I do the same thing with Java. The walkthrough includes open source code and unit tests. Mon, 28 Dec 2015 00:00:00 +0000 https://www.rathboma.com/2015/12/28/java-spark-tutorial.html https://www.rathboma.com/2015/12/28/java-spark-tutorial.html spark I implement a realistic pipeline in Spark as part of my series on Hadoop frameworks. The walkthrough includes open source code and a unit test. Mon, 14 Dec 2015 00:00:00 +0000 https://www.rathboma.com/2015/12/14/spark-scala-tutorial.html https://www.rathboma.com/2015/12/14/spark-scala-tutorial.html spark Apache Hive differs significantly from traditional relational databases like MySQL. I'll walk through some key differences in architecture and use cases to give a sense of when and why to use each technology. Tue, 08 Dec 2015 00:00:00 +0000 https://www.rathboma.com/2015/12/08/hive-vs-mysql.html https://www.rathboma.com/2015/12/08/hive-vs-mysql.html hadoop How to implement some real world code with Scalding, a Scala wrapper for Cascading Tue, 20 Oct 2015 00:00:00 +0000 https://www.rathboma.com/2015/10/20/scalding-tutorial.html https://www.rathboma.com/2015/10/20/scalding-tutorial.html hadoop How to implement all sorts of custom functions for hive, including complex functions like 'sum()', 'count()', and 'explode()' Mon, 27 Jul 2015 21:44:12 +0000 https://www.rathboma.com/2015/07/27/ultimate-guide-to-writing-custom-functions-for-hive.html https://www.rathboma.com/2015/07/27/ultimate-guide-to-writing-custom-functions-for-hive.html hadoop How to use Java & Cascading to perform a left outer join. Includes a comparison with Hive, Pig, and Java Mapreduce Thu, 25 Jun 2015 00:00:00 +0000 https://www.rathboma.com/2015/06/25/real-world-hadoop-implementing-a-left-outer-join-in-java-with-cascading.html https://www.rathboma.com/2015/06/25/real-world-hadoop-implementing-a-left-outer-join-in-java-with-cascading.html hadoop A rundown of the common query engines for Hadoop, with some of their pros/cons Sun, 08 Jun 2014 00:00:00 +0000 https://www.rathboma.com/2014/06/08/sql-engines-for-hadoop.html https://www.rathboma.com/2014/06/08/sql-engines-for-hadoop.html hadoop A survey on data warehousing and Hadoop Deployments. Tell me what problems are you having - I'd love to help you fix them. Tue, 21 Jan 2014 00:00:00 +0000 https://www.rathboma.com/2014/01/21/data-warehousing-and-hadoop-2014-survey.html https://www.rathboma.com/2014/01/21/data-warehousing-and-hadoop-2014-survey.html personal How to read files from hdfs using Java and Scala, even if they are compressed, using any FileSystem implementation. Sat, 28 Dec 2013 00:00:00 +0000 https://www.rathboma.com/2013/12/28/Reading-data-from-HDFS-even-if-it-is-compressed.html https://www.rathboma.com/2013/12/28/Reading-data-from-HDFS-even-if-it-is-compressed.html hadoop A step-by-step tutorial for writing your first map reduce with Python and Hadoop Streaming. Sun, 17 Nov 2013 00:00:00 +0000 https://www.rathboma.com/2013/11/17/python-map-reduce-on-hadoop-a-beginners-tutorial.html https://www.rathboma.com/2013/11/17/python-map-reduce-on-hadoop-a-beginners-tutorial.html hadoop How to use Scala & Scoobi to perform a left outer join. Includes a comparision with Hive, Pig, and Java Mapreduce Sun, 03 Nov 2013 00:00:00 +0000 https://www.rathboma.com/2013/11/03/real-world-hadoop-implementing-a-left-outer-join-with-scoobi.html https://www.rathboma.com/2013/11/03/real-world-hadoop-implementing-a-left-outer-join-with-scoobi.html hadoop a walk-through (with code) of writing user defined functions (UDFs) for Apache Hive. There are two APIs, so my walkthrough will include examples for both. Sat, 10 Aug 2013 00:00:00 +0000 https://www.rathboma.com/2013/08/10/guide-to-writing-hive-udfs.html https://www.rathboma.com/2013/08/10/guide-to-writing-hive-udfs.html hadoop A compilation of books for Hadoop and related projects (hive, pig, hbase, etc). Fri, 31 May 2013 00:00:00 +0000 https://www.rathboma.com/2013/05/31/hadoop-resources-books.html https://www.rathboma.com/2013/05/31/hadoop-resources-books.html hadoop What is Hadoop - A high level overview of Hadoop without going in to too much technical detail. Wed, 17 Apr 2013 00:00:00 +0000 https://www.rathboma.com/2013/04/17/what-is-hadoop.html https://www.rathboma.com/2013/04/17/what-is-hadoop.html hadoop How to use Pig to perform a left outer join. Includes a comparison with Hive and regular Java Map Reduce. Sun, 07 Apr 2013 00:00:00 +0000 https://www.rathboma.com/2013/04/07/real-world-hadoop-implementing-a-left-outer-join-in-pig.html https://www.rathboma.com/2013/04/07/real-world-hadoop-implementing-a-left-outer-join-in-pig.html hadoop blog post by Matthew Rathbone Wed, 27 Feb 2013 00:00:00 +0000 https://www.rathboma.com/2013/02/27/simple-check-in-now-has-more-features.html https://www.rathboma.com/2013/02/27/simple-check-in-now-has-more-features.html personal How to use hive to perform a left outer join. Includes a comparison with performing the same query in java map reduce. Wed, 20 Feb 2013 00:00:00 +0000 https://www.rathboma.com/2013/02/20/real-world-hadoop-implementing-a-left-outer-join-in-hive.html https://www.rathboma.com/2013/02/20/real-world-hadoop-implementing-a-left-outer-join-in-hive.html hadoop Thu, 14 Feb 2013 00:00:00 +0000 https://www.rathboma.com/2013/02/14/an-excellent-hadoop-resource-the-hadoop-weekly-newsletter.html https://www.rathboma.com/2013/02/14/an-excellent-hadoop-resource-the-hadoop-weekly-newsletter.html hadoop Sat, 09 Feb 2013 00:00:00 +0000 https://www.rathboma.com/2013/02/09/real-world-hadoop-implementing-a-left-outer-join-in-hadoop-map-reduce.html https://www.rathboma.com/2013/02/09/real-world-hadoop-implementing-a-left-outer-join-in-hadoop-map-reduce.html unclassified An exciting feature of sublime text 2 I never knew about. Sat, 26 Jan 2013 00:00:00 +0000 https://www.rathboma.com/2013/01/26/auto-indenting-text-in-sublime-text-2.html https://www.rathboma.com/2013/01/26/auto-indenting-text-in-sublime-text-2.html programming A constantly expanding list of 12+ hadoop frameworks, with code examples and documentation links Sat, 05 Jan 2013 00:00:00 +0000 https://www.rathboma.com/2013/01/05/a-quick-guide-to-hadoop-map-reduce-frameworks.html https://www.rathboma.com/2013/01/05/a-quick-guide-to-hadoop-map-reduce-frameworks.html hadoop Fri, 07 Dec 2012 00:00:00 +0000 https://www.rathboma.com/2012/12/07/do-you-spend-hours-formatting-word-documents.html https://www.rathboma.com/2012/12/07/do-you-spend-hours-formatting-word-documents.html office step-by-step guide to formatting references in APA, with screenshots. Fri, 09 Nov 2012 00:00:00 +0000 https://www.rathboma.com/2012/11/09/how-to-format-references-in-apa-style-using-microsoft-word-2011-for-mac-osx.html https://www.rathboma.com/2012/11/09/how-to-format-references-in-apa-style-using-microsoft-word-2011-for-mac-osx.html office Fri, 09 Nov 2012 00:00:00 +0000 https://www.rathboma.com/2012/11/09/how-to-do-find-and-replace-within-currently-selected-text-in-microsoft-word-2011-for-mac-osx.html https://www.rathboma.com/2012/11/09/how-to-do-find-and-replace-within-currently-selected-text-in-microsoft-word-2011-for-mac-osx.html office A useful Sublime Text 2 plugin for scala users Wed, 31 Oct 2012 00:00:00 +0000 https://www.rathboma.com/2012/10/31/adding-scaladocs-to-a-scala-function-in-sublime-text-2.html https://www.rathboma.com/2012/10/31/adding-scaladocs-to-a-scala-function-in-sublime-text-2.html programming a walkthrough and code for building games with scala using libgdx and box2d. Mon, 22 Oct 2012 00:00:00 +0000 https://www.rathboma.com/2012/10/22/game-programming-in-scala-with-libgdx-and-box2d.html https://www.rathboma.com/2012/10/22/game-programming-in-scala-with-libgdx-and-box2d.html programming Sat, 28 Apr 2012 00:00:00 +0000 https://www.rathboma.com/2012/04/28/simple-check-in-unknown-error-bug-is-fixed.html https://www.rathboma.com/2012/04/28/simple-check-in-unknown-error-bug-is-fixed.html android Sat, 14 Apr 2012 00:00:00 +0000 https://www.rathboma.com/2012/04/14/one-touch-check-ins-for-foursquare.html https://www.rathboma.com/2012/04/14/one-touch-check-ins-for-foursquare.html android A hidden feature of sublime text 2 that you can't possibly live without. Sun, 18 Mar 2012 00:00:00 +0000 https://www.rathboma.com/2012/03/18/scrolling-past-the-end-of-a-file-in-sublime-text-2.html https://www.rathboma.com/2012/03/18/scrolling-past-the-end-of-a-file-in-sublime-text-2.html programming Fri, 09 Mar 2012 00:00:00 +0000 https://www.rathboma.com/2012/03/09/tips-for-using-cdhs-hadoop-distribution-with-amazons-s3.html https://www.rathboma.com/2012/03/09/tips-for-using-cdhs-hadoop-distribution-with-amazons-s3.html hadoop a step-by-step walkthrough for adding custom page number layouts to Microsoft Word using section breaks and a lot of clicking Fri, 16 Dec 2011 00:00:00 +0000 https://www.rathboma.com/2011/12/16/how-to-have-page-numbers-start-on-page-3-using-ms-word-2011-for-mac.html https://www.rathboma.com/2011/12/16/how-to-have-page-numbers-start-on-page-3-using-ms-word-2011-for-mac.html office Sat, 26 Nov 2011 00:00:00 +0000 https://www.rathboma.com/2011/11/26/geopop-my-first-android-game.html https://www.rathboma.com/2011/11/26/geopop-my-first-android-game.html android Thu, 27 Oct 2011 00:00:00 +0000 https://www.rathboma.com/2011/10/27/link-beginning-android-game-development-book.html https://www.rathboma.com/2011/10/27/link-beginning-android-game-development-book.html android an alternative setup for deploying jars to hive. Thu, 20 Oct 2011 00:00:00 +0000 https://www.rathboma.com/2011/10/20/adding-jars-to-hive-without-using-add-jar.html https://www.rathboma.com/2011/10/20/adding-jars-to-hive-without-using-add-jar.html hadoop Tue, 04 Oct 2011 00:00:00 +0000 https://www.rathboma.com/2011/10/04/the-start-of-an-online-puzzle-community.html https://www.rathboma.com/2011/10/04/the-start-of-an-online-puzzle-community.html personal Tue, 20 Sep 2011 00:00:00 +0000 https://www.rathboma.com/2011/09/20/some-fun-hadoop-and-hive-bugs.html https://www.rathboma.com/2011/09/20/some-fun-hadoop-and-hive-bugs.html hadoop Tue, 19 Apr 2011 00:00:00 +0000 https://www.rathboma.com/2011/04/19/a-warning-about-apache-thrifts-asynchronous-clients.html https://www.rathboma.com/2011/04/19/a-warning-about-apache-thrifts-asynchronous-clients.html programming Mon, 28 Feb 2011 00:00:00 +0000 https://www.rathboma.com/2011/02/28/it-has-been-a-good-week.html https://www.rathboma.com/2011/02/28/it-has-been-a-good-week.html personal Fri, 25 Feb 2011 00:00:00 +0000 https://www.rathboma.com/2011/02/25/kindle-store-prices.html https://www.rathboma.com/2011/02/25/kindle-store-prices.html personal Wed, 29 Dec 2010 00:00:00 +0000 https://www.rathboma.com/2010/12/29/nullpointerexception-when-performing-a-join-with-hadoop-hive-050.html https://www.rathboma.com/2010/12/29/nullpointerexception-when-performing-a-join-with-hadoop-hive-050.html hadoop Sun, 07 Nov 2010 00:00:00 +0000 https://www.rathboma.com/2010/11/07/remember-stuff-easier.html https://www.rathboma.com/2010/11/07/remember-stuff-easier.html personal Sat, 09 Oct 2010 00:00:00 +0000 https://www.rathboma.com/2010/10/09/mini-squadron-my-favorite-android-game-of-the-moment.html https://www.rathboma.com/2010/10/09/mini-squadron-my-favorite-android-game-of-the-moment.html android There are some gotchas when serializing time to json and back. I outline a few Tue, 28 Sep 2010 00:00:00 +0000 https://www.rathboma.com/2010/09/28/ruby-json-serialization-sucks-for-time-objects.html https://www.rathboma.com/2010/09/28/ruby-json-serialization-sucks-for-time-objects.html programming Implementing quicksort (in place) using C, and recursively using Scheme / Lisp Tue, 10 Nov 2009 00:00:00 +0000 https://www.rathboma.com/2009/11/10/operation-algorithm-quicksort-in-c-and-scheme.html https://www.rathboma.com/2009/11/10/operation-algorithm-quicksort-in-c-and-scheme.html programming Wed, 03 Dec 2008 00:00:00 +0000 https://www.rathboma.com/2008/12/03/be-careful-of-data-obtained-from-bloomberg-using-rtd-or-bdp-functions.html https://www.rathboma.com/2008/12/03/be-careful-of-data-obtained-from-bloomberg-using-rtd-or-bdp-functions.html office