[Apache Spark] vs. [Spark Core] Forum? {SOLVED}

General
1

(ported from mailing list question)

What is the purpose of the randomly assigned SPARK_WORKER_PORT

from the documentation it sais to “join a cluster”, but its not clear to me
how a random port could be used to communicate with other members of a
spark  pool.

Thanks !

2

Hi @jayunit100,

It sounds like you’re asking about http://spark.apache.org/ , and not the Spark Core, is that right? :slight_smile: Just a heads up that this isn’t the forum for Apache Spark, but we can try to help anyway.

I’m guessing the SPARK_WORKER_PORT is used by the workers to communicate with eachother somehow, and it being random prevents collisions and makes it easier to run simulated clusters on a single machine. It also probably provides some minimal security by obscurity, I found a page talking a little bit about this here: http://spark.apache.org/docs/latest/spark-standalone.html

Thanks,
David

4 Likes
3

ha ! oops. yes your right, im in the wrong forum.

3 Likes