Skip to content

Using Polynote with Python

For now, Python is the major non-JVM-based language you'll be using with Polynote. Polynote uses Jep to execute Python code within the JVM. Jep does most of the heavy-lifting when it comes to Python interop.

Most Python code should work out of the box on Polynote, without needing anything special. Please let us know if you run into any problems with Python code that works well in Jupyter or the Python REPL.

Python dependencies

When you specify a Python dependency in the configuration, Polynote creates a virtual environment scoped to your notebook.

This virtual environment is reused on subsequent runs of the notebook, unless your dependencies change or you explicitly bust the cache.

Additionally, the virtual environments are isolated from each other but not from the system, since Polynote specifies the --system-site-packages flag when creating the environment.

Experimental: PySpark And Dependencies

Polynote attempts to add the dependencies you specify to Spark by downloading their zip files and adding them to the Spark context if it exists.

This means that your Python dependencies should be shipped to your executors (and available) to your code that runs there!

Sharing between Python and Scala

Our goals right now are to support a few, key use-cases with a focus on sharing from Scala to Python, such as plotting data generated in Scala with matplotlib, or using Scala-generated data with tensorflow and scikit-learn. We've found that the interop between Python and Scala can be very powerful even if it is limited to these simple cases.

Tip

If you're going to be moving back and forth between Python and Scala a lot, we highly recommend reading how Jep works.

Here are a few important points to keep in mind when sharing between Python and Scala:

  • Jep handles the conversion from Scala -> Python.
    • It converts primitives and strings into brand-new Python primitives and strings.
    • An object of any other type is wrapped as a PyJObject, which is an interface allowing Python to directly access that objects attributes. Note that in this case, nothing is copied - PyJObject holds a reference to the underlying JVM object.
    • Note that Jep is based on Java, not Scala. This means that when it wraps a Scala object as a PyJObject, you won't get Scala sugar - things like multiple parameter lists, implicits, etc. - when you work with it in Python. This can limit your ability to use a lot of super-scala-stuff with Python.
  • Jep handles conversion from Python -> Scala and Polynote adds a little bit of sugar on top.
    • Similar to the other way round, Jep automatically converts primitives and strings into brand-new JVM primitives and strings.
    • Additionally, Jep supports some other conversions such as Python dict to java.util.HashMap
    • Polynote will retrieve an object of any other type as a PyObject. Similar to PyJObject, a PyObject wraps a pointer to a Python object.

Going from Python to Scala

As previously mentioned, Polynote has some extra sugar for handling certain types of Python objects as part of the PythonObject API, which helps with things like using Scala-specific syntax on these data types as well as visualization.

For example, if a user wanted to iterate over a Python List (which gets converted automatically to a java.util.ArrayList) using for-comprehension, they would get a runtime error. The PythonObject API offers the asScalaList method, which handles the conversion to return a List[PythonObject], as demonstrated in the below example.

Python Cell

ids = [1234, 5678, 9012, 3456]
typs = ['Number', 'Number', 'Number', 'Number']

Scala Cell

case class ResultItem(id: Int, typ: String)

val resultData = for {
  i <- ids.asScalaList
  t <- typs.asScalaList
} yield ResultItem(i.as[Integer], t.as[String])

More Details on the PythonObject API

If you'd like to view the entire PythonObject API, you can do so here.

Note that these implementation details may change and while we'll work hard to update this information we can't guarantee that it won't get out-of-date. Of course, feel free to drop us a line if you think that's the case!