View llmdeploy on Snyk Open Source Advisor

Package Health Score

61 / 100

security
No known security issues
popularity
Small
maintenance
Inactive
community
Active

Explore Similar Packages

ai
57

Popularity

Small

GitHub Stars: 4.57K
Forks: 416
Contributors: 80

Direct Usage Popularity

Based on project statistics from the GitHub repository for the PyPI package llmdeploy, we found that it has been starred 4,570 times.

Security

No known security issues

Security and license risk for latest version

Release Date: Jun 29, 2023
Direct Vulnerabilities: 0
C
0
H
0
M
0
L
Indirect Vulnerabilities: 0
C
0
H
0
M
0
L
License Risk: 0
H
0
M
0
L

All security vulnerabilities belong to production dependencies of direct and indirect packages.

License

Apache-2.0

Security Policy

We found a way for you to contribute to the project! Looks like llmdeploy is missing a security policy.

You can connect your project's repository to Snyk to stay up to date on security alerts and receive automatic fix pull requests.

Keep your project free of vulnerabilities with Snyk

Maintenance

Inactive

Commit Frequency

Open Issues: 287
Open PR: 27
Last Release: 1 year ago
Last Commit: 19 days ago

Further analysis of the maintenance status of llmdeploy based on released PyPI versions cadence, the repository activity, and other data points determined that its maintenance is Inactive.

An important project maintenance signal to consider for llmdeploy is that it hasn't seen any new versions released to PyPI in the past 12 months, and could be considered as a discontinued project, or that which receives low attention from its maintainers.

As a healthy sign for on-going project maintenance, we found that the GitHub repository had at least 1 pull request or issue interacted with by the community.

Community

Active

Readme: Yes
Contributing.md: Yes
Code of Conduct: No
Contributors: 80
Funding: No

With more than 10 contributors for the llmdeploy repository, this is possibly a sign for a growing and inviting community.

We found a way for you to contribute to the project! Looks like llmdeploy is missing a Code of Conduct.

Embed Package Health Score Badge

Keep your project healthy

Check your requirements.txt

Ensure all the packages you're using are healthy and well-maintained

Snyk Vulnerability Scanner

Get health score & security insights directly in your IDE

Package

Python Versions Compatibility: ==3.10.*, ==3.8.*, ==3.9.*

Age: 1 year
Latest Release: 1 year ago
Dependencies: N/A
Versions: 1
Maintainers: 1
Wheels: OS Independent

Readme

OpenMMLab website ^HOT OpenMMLab platform ^{TRY IT OUT}

English | 简体中文

Introduction

Installation

Below are quick steps for installation:

conda create -n open-mmlab python=3.8
conda activate open-mmlab
git clone https://github.com/open-mmlab/llmdeploy.git
cd llmdeploy
pip install -e .

Quick Start

Build

Pull docker image openmmlab/llmdeploy:base and build llmdeploy libs in its launched container

mkdir build &amp;&amp; cd build
../generate.sh
make -j$(nproc) &amp;&amp; make install

Serving LLaMA

Weights for the LLaMA models can be obtained from by filling out this form

Run one of the following commands to serve a LLaMA model on NVIDIA GPU server:

python3 llmdeploy/serve/fastertransformer/deploy.py llama-7B /path/to/llama-7b llama \
    --tokenizer_path /path/to/tokenizer/model
bash workspace/service_docker_up.sh --lib-dir $(pwd)/build/install/backends/fastertransformer

13B

python3 llmdeploy/serve/fastertransformer/deploy.py llama-13B /path/to/llama-13b llama \
    --tokenizer_path /path/to/tokenizer/model --tp 2
bash workspace/service_docker_up.sh --lib-dir $(pwd)/build/install/backends/fastertransformer

33B

python3 llmdeploy/serve/fastertransformer/deploy.py llama-33B /path/to/llama-33b llama \
    --tokenizer_path /path/to/tokenizer/model --tp 4
bash workspace/service_docker_up.sh --lib-dir $(pwd)/build/install/backends/fastertransformer

65B

python3 llmdeploy/serve/fastertransformer/deploy.py llama-65B /path/to/llama-65b llama \
    --tokenizer_path /path/to/tokenizer/model --tp 8
bash workspace/service_docker_up.sh --lib-dir $(pwd)/build/install/backends/fastertransformer

Serving Vicuna

python3 -m pip install fschat
python3 -m fastchat.model.apply_delta \
  --base-model-path /path/to/llama-7b \
  --target-model-path /path/to/vicuna-7b \
  --delta-path lmsys/vicuna-7b-delta-v1.1

python3 llmdeploy/serve/fastertransformer/deploy.py vicuna-7B /path/to/vicuna-7b hf
bash workspace/service_docker_up.sh --lib-dir $(pwd)/build/install/backends/fastertransformer

13B

python3 -m pip install fschat
python3 -m fastchat.model.apply_delta \
  --base-model-path /path/to/llama-13b \
  --target-model-path /path/to/vicuna-13b \
  --delta-path lmsys/vicuna-13b-delta-v1.1

python3 llmdeploy/serve/fastertransformer/deploy.py vicuna-13B /path/to/vicuna-13b hf
bash workspace/service_docker_up.sh --lib-dir $(pwd)/build/install/backends/fastertransformer

Inference with Command Line Interface

python3 llmdeploy/serve/client.py {server_ip_addresss}:33337 1

Inference with Web UI

python3 llmdeploy/webui/app.py {server_ip_addresss}:33337 model_name

User Guide

Quantization

In fp16 mode, kv_cache int8 quantization can be enabled, and a single card can serve more users. First execute the quantization script, and the quantization parameters are stored in the weight directory transformed by deploy.py. Then adjust config.ini

use_context_fmha changed to 0, means off
quant_policy is set to 4. This parameter defaults to 0, which means it is not enabled

Contributing

We appreciate all contributions to LLMDeploy. Please refer to CONTRIBUTING.md for the contributing guideline.

Acknowledgement

FasterTransformer

License

This project is released under the Apache 2.0 license.

FAQs

What is llmdeploy?

triton inference service of llama. Visit Snyk Advisor to see a full health score report for llmdeploy, including popularity, security, maintenance & community analysis.

Is llmdeploy popular?

The python package llmdeploy receives a total of 10 weekly downloads. As such, llmdeploy popularity was classified as small. Visit the popularity section on Snyk Advisor to see the full health analysis.

Is llmdeploy well maintained?

We found indications that llmdeploy is an Inactive project. See the full package health analysis to learn more about the package maintenance status.

Is llmdeploy safe to use?

The python package llmdeploy was scanned for known vulnerabilities and missing license, and no issues were found. Thus the package was deemed as safe to use. See the full health analysis review.

Last updated on 21 November-2024, at 10:34 (UTC).

Build a secure application checklist

Select a recommended open source package

Minimize your risk by selecting secure & well maintained open source packages

DONE

Scan your app for vulnerabilities

Scan your application to find vulnerabilities in your: source code, open source dependencies, containers and configuration files

SCAN NOW

Fix identified vulnerabilities

Easily fix your code by leveraging automatically generated PRs

AUTO FIX

Monitor for new issues

New vulnerabilities are discovered every day. Get notified if your application is affected

MONITOR APP

Package Health Score

Explore Similar Packages

Direct Usage Popularity

Security and license risk for latest version

Commit Frequency

Embed Package Health Score Badge

Keep your project healthy

Check your requirements.txt

Snyk Vulnerability Scanner

Readme

Introduction

Installation

Quick Start

Build

Serving LLaMA

Serving Vicuna

Inference with Command Line Interface

Inference with Web UI

User Guide

Quantization

Contributing

Acknowledgement

License

FAQs

What is llmdeploy?

Is llmdeploy popular?

Is llmdeploy well maintained?

Is llmdeploy safe to use?

Build a secure application checklist

Select a recommended open source package

Scan your app for vulnerabilities

Source Code

Open Source

Container

Config

Fix identified vulnerabilities

Monitor for new issues