TensorFlowOnSpark

Why we included this project

TensorFlowOnSpark runs distributed TensorFlow training and inference directly on existing Apache Spark and Hadoop clusters. That means a team that already manages Spark for data engineering can start doing deep learning without standing up a separate scheduler or moving data to another platform. Yahoo built it internally for large-scale model training on its own Hadoop grid, and the API is designed so that most existing TensorFlow programs need only a handful of lines changed to run across GPU and CPU executors. It handles synchronous and asynchronous training, model and data parallelism, and lets you feed data either from HDFS via TensorFlow's readers or from Spark RDDs through a DataFeed helper. If your stack is already Spark-centric, this is a pragmatic way to keep model training in the same cluster as your data processing.