#INSTALL SPARK ON WINDOWS INSTALL#

JAVA_HOME: usually you already set this variable when you install java but it is better to verify that exist and is correct.

To fix this inconvenient I set this variable to the folder containing the winutils.exe file

HADOOP_HOME: even when Spark can run without Hadoop, the version I downloaded is prebuilt for Hadoop 2.6 and looks in the code for it.

I was getting Java Heap Memory problems with the default values and this fixed this problem.

_JAVA_OPTION: I set this variable to the value showed in the figure below.

This is also crucial in order to run some commands without problems using the command prompt. Git client download (I hope you don’t get stuck in this step)Įxtract the folder containing the file winutils.exe to any location of your preference. If you know another link where we can found this file you can share it with us. In order to make my trip still longer, I had to install Gitto be able to download the 32-bits winutils.exe. This was the critical point for me, because I downloaded one version and did not work until I realized that there are 64-bits and 32-bits versions of this file. Start –> All apps –> Java –> Check For Updatesįeel free also to download the source code and make your own build if you feel comfortable with it.Įxtract the files to any location in your drive with enough permissions for your user.

In this case I just updated my java version as follows: Install or update Javaįor any application that uses the Java Virtual Machine is always recommended to install the appropriate java version. We are going to install Spark 1.6.0 as standalone in a computer with a 32-bit Windows 10 installation (my very old laptop). Spark runs on Hadoop, Mesos, in the cloud or as standalone. It is possible to write Spark applications using Java, Python, Scala and R, and it comes with built-in libraries to work with structure data ( Spark SQL), graph computation ( GraphX), machine learning ( MLlib) and streaming ( Spark Streaming). A few words about Apache SparkĪpache Spark is making a lot of noise in the IT world as a general engine for large-scale data processing, able to run programs up to 100x faster than Hadoop MapReduce, thanks to its in-memory computing capabilities. The new version of these VMs come with Spark ready to use.

If you really want to build a serious prototype, I strongly recommend to install one of the virtual machines I mentioned in this post a couple of years ago: Hadoop self-learning with pre-configured Virtual Machines or to spend some money in a Hadoop distribution on the cloud.

#INSTALL SPARK ON WINDOWS HOW TO#

This post is to help people to install and run Apache Spark in a computer with window 10 (it may also help for prior versions of Windows or even Linux and Mac OS systems), and want to try out and learn how to interact with the engine without spend too many resources.

Install spark on windows