records

Type Members

trait AccumulatorMetrics extends Metrics

A Metrics implementation using com.swoop.spark.accumulators.ByKeyAdditiveAccumulator.
trait BuildContext[Ctx] extends HasJobContext[Ctx] with HasFeatures with HasDetails with ThrowsErrors

Convenience trait that combines the main services exposed by RecordBuilder.
Convenience trait that combines the main services exposed by RecordBuilder.
Ctx
the job context

See also
Build Context
trait BuildsRecords[Rec <: Record[_, _]] extends Serializable

Topmost trait for record builders.
case class Cause(message: String, stack: Seq[StackElement] = Seq.empty) extends Product with Serializable

A simple, serializable exception cause that keeps the exception message and the stack trace.
abstract class CompositeRecordBuilder[BuildInput, RecordInput, Rec <: Record[_, _], Ctx <: JobContext] extends RecordBuilderLike[Rec, Ctx]

Base class for record builders that produce multiple records for one input.
Base class for record builders that produce multiple records for one input.
BuildInput
The input type for the builder.
RecordInput
The input type records are built from.
Rec
The record type.
Ctx
The job context type

See also
NestedRecordBuilder
Composite Builders
trait DriverContext extends Serializable with Metrics with Flight

Shared behaviors for the build context that integrates with Spark context.
Shared behaviors for the build context that integrates with Spark context.

See also
Build Context
trait FeatureMask extends AnyRef

Trait for types that have a feature mask.
trait Features extends AnyRef

Definitions for the meaning of bits in the features bit mask of com.swoop.spark.records.Record.
Definitions for the meaning of bits in the features bit mask of com.swoop.spark.records.Record.
The least significant 16 bits are reserved for framework use. The next 15 bits are available for application use. The most significant bit is reserved for framework use also.
class FlatRecordEnvironment extends RecordEnvironment

A flat record environment.
trait Flight extends AnyRef

Trait identifying flights: one or more related Spark jobs.
trait HasDetails extends AnyRef

Trait for a builder that aggregates details, which will become issues.
Trait for a builder that aggregates details, which will become issues.
Under the covers, details are Throwable instances: either exceptions thrown by the application code or ThrowableMessage instances that may be thrown and caught or simply constructed using one of the convenience methods exposed by this trait.
trait HasFeatures extends AnyRef

Trait for a builder that aggregates features.
trait HasJobContext[Ctx] extends AnyRef

Trait for a builder with a job context.
case class Issue(category: Int, message: String, causes: Option[Seq[Cause]] = None, id: Option[Int] = None, details: Option[String] = None) extends Product with Serializable

An issue is a container for diagnostic information including both simple fields and an optional sequence of causes.
An issue is a container for diagnostic information including both simple fields and an optional sequence of causes.
category
The category of the issue. It matches the with the values coming from the different types of Issue$.ThrowableMessageType featureMask values.
message
The message related to the issue.
causes
The causes of this issue.
id
The issue ID.
details
Optional additional details for the issue. For root cause analysis purposes it is useful to to split messages we want to associate with records into a stable portion (the issue message) and a variable portion (the issue details). This enables stable grouping and counting.

See also
Issue$.messageWithDetails
class IssuesDataFrame extends RootCauseAnalysisOps

Provides root cause analysis operations at the issues level.
trait JobContext extends Serializable with Metrics with Flight

Shared behaviors for the build context that provides services to builders but does not work directly with Spark context APIs, allowing fast Spark-less testing.
Shared behaviors for the build context that provides services to builders but does not work directly with Spark context APIs, allowing fast Spark-less testing.

See also
Build Context
trait Metrics extends AnyRef

Shared trait defining metrics collection services
abstract class NestedRecordBuilder[Input, Data <: Product, Rec <: Record[Data, _], Ctx <: JobContext] extends RecordBuilder[Input, Data, Rec, Ctx]

Base class for nested record builders of single records.
Base class for nested record builders of single records.
Input
The source input
Data
The record data
Rec
The record
Ctx
The job context
class PartitionedRecordEnvironment extends RecordEnvironment

A partitioned error environment.
implicit class RCARichDataFrame extends AnyRef

Implicit operations on Spark Records dataframes.
Implicit operations on Spark Records dataframes.
Most operations can be applied equally at the record level or at the level of unwound issues.

See also
RootCauseAnalysis
RootCauseAnalysisOps
trait Record[A <: Product, Source] extends Product with Serializable

A Spark Record is an envelope around data.
A Spark Record is an envelope around data.
A
The data stored in the record
Source
The input type

See also
Spark Records documentation
abstract class RecordBuilder[Input, Data <: Product, Rec <: Record[Data, _], Ctx <: JobContext] extends RecordBuilderLike[Rec, Ctx] with BuildContext[Ctx]

Base class for record builders of single records.
Base class for record builders of single records.
Input
The source input
Data
The record data
Rec
The record
Ctx
The job context
abstract class RecordBuilderLike[Rec <: Record[_, _], Ctx <: JobContext] extends BuildsRecords[Rec] with HasJobContext[Ctx] with ThrowsErrors

Base class for single and composite record builders.
Base class for single and composite record builders.
Rec
The record
Ctx
The job context
trait RecordEnvironment extends Serializable

Base trait for the implicits that provide hints about how records are persisted.
Base trait for the implicits that provide hints about how records are persisted.

Annotations
@implicitNotFound( ... )
class RecordsDataFrame extends RootCauseAnalysisOps

Provides root cause analysis operations at the record level.
implicit class RecordsRichDataset[A <: Product, Rec] extends Serializable

Implicit operations on Spark Records datasets.
Implicit operations on Spark Records datasets.

See also
RootCauseAnalysisOps
implicit final class RecordsRichOption[A] extends AnyVal

Extends Option with record building sugar.
implicit final class RecordsRichTry[A] extends AnyVal

Extends scala.util.Try with record building sugar.
implicit final class RecordsRichTryOption[A] extends AnyVal

Extends scala.util.Try with optional values with record building sugar.
class RootCauseAnalysis extends RootCauseAnalysisOps

Provides root cause analysis operations at any level by automatically dispatching to RecordsDataFrame or IssuesDataFrame based on the schema of the provided dataframe.
trait RootCauseAnalysisOps extends Serializable

Base trait for root cause analysis operations.
class SimpleDriverContext extends DriverContext with AccumulatorMetrics

Basic implementation of DriverContext that uses AccumulatorMetrics and includes a convenience factory for creating job contexts.
Basic implementation of DriverContext that uses AccumulatorMetrics and includes a convenience factory for creating job contexts.

See also
Build Context
class SimpleJobContext extends JobContext with AccumulatorMetrics

Basic implementation of JobContext that uses AccumulatorMetrics.
Basic implementation of JobContext that uses AccumulatorMetrics.

Note
Spark accumulators can be used locally without registration.
See also
Build Context
case class StackElement(className: String, methodName: String, fileName: String, lineNumber: Int) extends Product with Serializable

A simple, serializable adapter for StackTraceElement.
case class ThrowableMessage(kind: ThrowableMessageType, msg: String, msgId: Int = 0, cause: Throwable = null) extends Throwable with FeatureMask with Product with Serializable

Base class for throwable diagnostic messages with IDs for quick big data processing.
Base class for throwable diagnostic messages with IDs for quick big data processing.
kind
the type of message (error, warning, info or debug)
msg
the message
msgId
the optional message ID
cause
the cause, if the message is being thrown as a result of an exception
trait ThrowsErrors extends AnyRef

Convenience trait for throwing ThrowableMessage.

Value Members

object FeatureMask
object Features extends Features
object FlatRecordEnvironment extends Serializable
object Issue extends ThrowsErrors with Serializable
object PartitionedRecordEnvironment extends Serializable
object Record extends Serializable
object RootCauseAnalysis extends Serializable
object SimpleDriverContext extends Serializable
object SimpleJobContext extends Serializable
implicit def optionalIssueSeq(issues: Seq[Issue]): Option[Seq[Issue]]

Convenience conversion to make issue saving during record creation easier.

package records

Type Members

trait AccumulatorMetrics extends Metrics

trait BuildContext[Ctx] extends HasJobContext[Ctx] with HasFeatures with HasDetails with ThrowsErrors

trait BuildsRecords[Rec <: Record[_, _]] extends Serializable

case class Cause(message: String, stack: Seq[StackElement] = Seq.empty) extends Product with Serializable

abstract class CompositeRecordBuilder[BuildInput, RecordInput, Rec <: Record[_, _], Ctx <: JobContext] extends RecordBuilderLike[Rec, Ctx]

trait DriverContext extends Serializable with Metrics with Flight

trait FeatureMask extends AnyRef

trait Features extends AnyRef

class FlatRecordEnvironment extends RecordEnvironment

trait Flight extends AnyRef

trait HasDetails extends AnyRef

trait HasFeatures extends AnyRef

trait HasJobContext[Ctx] extends AnyRef

case class Issue(category: Int, message: String, causes: Option[Seq[Cause]] = None, id: Option[Int] = None, details: Option[String] = None) extends Product with Serializable

class IssuesDataFrame extends RootCauseAnalysisOps

trait JobContext extends Serializable with Metrics with Flight

trait Metrics extends AnyRef

abstract class NestedRecordBuilder[Input, Data <: Product, Rec <: Record[Data, _], Ctx <: JobContext] extends RecordBuilder[Input, Data, Rec, Ctx]

class PartitionedRecordEnvironment extends RecordEnvironment

implicit class RCARichDataFrame extends AnyRef

trait Record[A <: Product, Source] extends Product with Serializable

abstract class RecordBuilder[Input, Data <: Product, Rec <: Record[Data, _], Ctx <: JobContext] extends RecordBuilderLike[Rec, Ctx] with BuildContext[Ctx]

abstract class RecordBuilderLike[Rec <: Record[_, _], Ctx <: JobContext] extends BuildsRecords[Rec] with HasJobContext[Ctx] with ThrowsErrors

trait RecordEnvironment extends Serializable

class RecordsDataFrame extends RootCauseAnalysisOps

implicit class RecordsRichDataset[A <: Product, Rec] extends Serializable

implicit final class RecordsRichOption[A] extends AnyVal

implicit final class RecordsRichTry[A] extends AnyVal

implicit final class RecordsRichTryOption[A] extends AnyVal

class RootCauseAnalysis extends RootCauseAnalysisOps

trait RootCauseAnalysisOps extends Serializable

class SimpleDriverContext extends DriverContext with AccumulatorMetrics

class SimpleJobContext extends JobContext with AccumulatorMetrics

case class StackElement(className: String, methodName: String, fileName: String, lineNumber: Int) extends Product with Serializable

case class ThrowableMessage(kind: ThrowableMessageType, msg: String, msgId: Int = 0, cause: Throwable = null) extends Throwable with FeatureMask with Product with Serializable

trait ThrowsErrors extends AnyRef

Value Members

object FeatureMask

object Features extends Features

object FlatRecordEnvironment extends Serializable

object Issue extends ThrowsErrors with Serializable

object PartitionedRecordEnvironment extends Serializable

object Record extends Serializable

object RootCauseAnalysis extends Serializable

object SimpleDriverContext extends Serializable

object SimpleJobContext extends Serializable

implicit def optionalIssueSeq(issues: Seq[Issue]): Option[Seq[Issue]]

Inherited from AnyRef

Inherited from Any

Ungrouped