Skip to content

Custom Input Partitioning #691

@kngine-rash

Description

@kngine-rash

I'm trying to use newAPIHadoopFile to implement custom InputFormat and RecordReader classes instead of using the default ones provided by Spark.
I can do the same in Java but am unable to find the correct way of achieving this using Mobius.

For reference, these are the classes and interfaces I want to override

java.lang.Object
org.apache.hadoop.mapreduce.InputFormat<K,V>
org.apache.hadoop.mapreduce.lib.input.FileInputFormat<K,V>

and Interface RecordReader<K,V>

https://hadoop.apache.org/docs/r2.4.1/api/org/apache/hadoop/mapreduce/RecordReader.html

https://hadoop.apache.org/docs/stable/api/index.htmlorg/apache/hadoop/mapreduce/lib/input/FileInputFormat.html

In Mobius, trying to use a custom InputFormat class throws a 'ClassNotFound' exception. I'm guessing it only excepts classes provided by Hadoop.

Does Mobius support custom partitioning ?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions