Link Search Menu Expand Document

Roadmap 2021

January 29, 2021 by Alex Chermenin


Table of contents

  1. General vision
  2. Current focus
    1. Schemed collection support
    2. Connectors development
    3. CEP improvements
    4. Testing
  3. Next steps
    1. REPL
  4. How to get involved?

The following roadmap gives an overview of what we are currently working on and what we want to tackle next. This helps potential contributors understand the current status of your project and where it’s going next, as well as giving a chance to be part of the planning.

General vision

Shift the approach for data pipelines from schemaless (PCollection) to the schemed side (DataFrame) and SQL-like queries to be more productive in data processing and to be more similar to other tools and make it easier to move from such tools to Kio and Beam

Current focus

Schemed collection support

What: We want to create a wrapper for the schemed collections called DataFrame and implement SQL-like methods for data processing (similar to Spark or Pandas).

Status: in progress

Connectors development

What: It’s necessary to cover all connectors implemented in Beam at least and create a good documentation for that.

Status: in progress

CEP improvements

What: It would be great to extend the CEP library to support more complex patterns to match.

Status: planning

Testing

What: We want to increase code coverage by tests and after that keep in this certain level of coverage.

Status: planning

Next steps

REPL

What: The REPL (maybe will be called kio-shell) will be a great tool to analyze data interactively using the command line without having to write any code in an IDE and compile it before running.

Status: not planned yet

How to get involved?

Let us know in Gitter or in the Slack channels (#beam-kotlin in ASF or #kotlin-beam in Kotlin) if you are interested in taking a stab at leading the development of one of these features or just want to help us in the development process.

And of course, feel free to submit new issues and pull requests to our repository on GitHub: https://github.com/chermenin/kio. More information you can find in the Contribute section.