vdk-lineage

v0.3.1431637373

VDK Lineage plugin collects lineage (input -> job -> output) information and send it to a pre-configured destination. For more information about how to use this package see README

Latest version published 3 months ago

License: Apache-2.0

PyPI

GitHub

Package Health Score

71 / 100

security
Security review needed
popularity
Limited
maintenance
Healthy
community
Active

Popularity

Limited

GitHub Stars: 428
Forks: 57
Contributors: 40

Direct Usage Popularity

Based on project statistics from the GitHub repository for the PyPI package vdk-lineage, we found that it has been starred 428 times.

Security

Security review needed

Security and license risk for latest version

Release Date: Aug 29, 2024
Direct Vulnerabilities: 0
C
0
H
0
M
0
L
Indirect Vulnerabilities: 0
C
0
H
0
M
0
L
License Risk: 0
H
2
M
0
L

All security vulnerabilities belong to production dependencies of direct and indirect packages.

License

Apache-2.0

Security Policy

We found a way for you to contribute to the project! Looks like vdk-lineage is missing a security policy.

You can connect your project's repository to Snyk to stay up to date on security alerts and receive automatic fix pull requests.

Keep your project free of vulnerabilities with Snyk

Maintenance

Healthy

Commit Frequency

Open Issues: 225
Open PR: 16
Last Release: 3 months ago
Last Commit: 2 months ago

Further analysis of the maintenance status of vdk-lineage based on released PyPI versions cadence, the repository activity, and other data points determined that its maintenance is Healthy.

We found that vdk-lineage demonstrates a positive version release cadence with at least one new version released in the past 3 months.

As a healthy sign for on-going project maintenance, we found that the GitHub repository had at least 1 pull request or issue interacted with by the community.

Community

Active

Readme: Yes
Contributing.md: Yes
Code of Conduct: Yes
Contributors: 40
Funding: No

With more than 10 contributors for the vdk-lineage repository, this is possibly a sign for a growing and inviting community.

Embed Package Health Score Badge

Keep your project healthy

Check your requirements.txt

Ensure all the packages you're using are healthy and well-maintained

Snyk Vulnerability Scanner

Get health score & security insights directly in your IDE

Package

Python Versions Compatibility: ==3.10.*, ==3.11.*, ==3.12.*, ==3.7.*, ==3.8.*, ==3.9.*

Age: 3 years
Latest Release: 3 months ago
Dependencies: 5 Direct / 26 Total
Versions: 20
Maintainers: 1
Wheels: No

Readme

VDK Lineage

VDK Lineage plugin provides lineage data (input data -> job -> output data) information and send it to a pre-configured destination. The lineage data is send using OpenLineage standard

At POC level currently.

Currently, lineage data is collected

For each data job run/execution both start and end events including the status of the job (failed/succeeded)
For each execute query we collect input and output tables.

TODOs:

Collect status of the SQL query (failed, succeeded)
Create parent /child relationship between sql event and job run event to track them better (single job can have multiple queries)
Non-SQL lineage (ingest, load data,etc)
Extend support for all queries
provide more information using facets – op id, job version,
figure out how to visualize parent/child relationships in Marquez
Explore openlineage.sqlparser instead of sqllineage library as alternative

Usage

pip install vdk-lineage

And it will start collecting lineage from job and sql queries.

To send data using openlineage specify VDK_OPENLINEAGE_URL. For example:

export VDK_OPENLINEAGE_URL=http://localhost:5002
vdk marquez-server --start
vdk run some-job
# check UI for lineage
# stopping the server will delete any lineage data.
vdk marquez-server --stop

Build and testing

In order to build and test a plugin go to the plugin directory and use ../build-plugin.sh script to build it

vdk-lineage dependencies

openlineage-integration-common openlineage-python sqlparse vdk-core vdk-lineage-model

FAQs

What is vdk-lineage?

VDK Lineage plugin collects lineage (input -> job -> output) information and send it to a pre-configured destinat... Visit Snyk Advisor to see a full health score report for vdk-lineage, including popularity, security, maintenance & community analysis.

Is vdk-lineage popular?

The python package vdk-lineage receives a total of 719 weekly downloads. As such, vdk-lineage popularity was classified as limited. Visit the popularity section on Snyk Advisor to see the full health analysis.

Is vdk-lineage well maintained?

We found that vdk-lineage demonstrated a healthy version release cadence and project activity. It has a community of 40 open source contributors collaborating on the project. See the full package health analysis to learn more about the package maintenance status.

Is vdk-lineage safe to use?

While scanning the latest version of vdk-lineage, we found that a security review is needed. A total of 0 vulnerabilities or license issues were detected. See the full security scan results.

Last updated on 18 November-2024, at 22:27 (UTC).

Build a secure application checklist

Select a recommended open source package

Minimize your risk by selecting secure & well maintained open source packages

DONE

Scan your app for vulnerabilities

Scan your application to find vulnerabilities in your: source code, open source dependencies, containers and configuration files

SCAN NOW

Fix identified vulnerabilities

Easily fix your code by leveraging automatically generated PRs

AUTO FIX

Monitor for new issues

New vulnerabilities are discovered every day. Get notified if your application is affected

MONITOR APP