State
The context object provides astate
attribute that allows you to read and write key-value pairs.
The state is stored in database and can be accessed in any transform function in the pipeline.
Also you can access the state in the next executions of the pipeline.
State Callbacks
You can write your states by various conditions likesuccess
, failure
, now
.
Resources
PySpark Session
You can access the PySpark session using thepyspark
attribute of the resources
object.