Roadmap 2021
January 29, 2021 by Alex Chermenin
Table of contents
The following roadmap gives an overview of what we are currently working on and what we want to tackle next. This helps potential contributors understand the current status of your project and where it’s going next, as well as giving a chance to be part of the planning.
General vision
Shift the approach for data pipelines from schemaless (PCollection) to the schemed side (DataFrame) and SQL-like queries to be more productive in data processing and to be more similar to other tools and make it easier to move from such tools to Kio and Beam |
Current focus
Schemed collection support
What: We want to create a wrapper for the schemed collections called DataFrame and implement SQL-like methods for data processing (similar to Spark or Pandas).
Status: in progress
Connectors development
What: It’s necessary to cover all connectors implemented in Beam at least and create a good documentation for that.
Status: in progress
CEP improvements
What: It would be great to extend the CEP library to support more complex patterns to match.
Status: planning
Testing
What: We want to increase code coverage by tests and after that keep in this certain level of coverage.
Status: planning
Next steps
REPL
What: The REPL (maybe will be called kio-shell) will be a great tool to analyze data interactively using the command line without having to write any code in an IDE and compile it before running.
Status: not planned yet
How to get involved?
Let us know in Gitter or in the Slack channels (#beam-kotlin in ASF or #kotlin-beam in Kotlin) if you are interested in taking a stab at leading the development of one of these features or just want to help us in the development process.
And of course, feel free to submit new issues and pull requests to our repository on GitHub: https://github.com/chermenin/kio. More information you can find on the Community page.