tyler/python-requirements-txt-generator
1
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects 1 Releases Wiki Activity

Make into a WASM project #1

New Issue
Open
opened 2024-12-25 23:40:46 +00:00 by tyler · 1 comment
tyler commented 2024-12-25 23:40:46 +00:00
Owner
Copy Link

cc @pyrate

This is another project of mine I just 'finished', not directly related to the v2 of the kjv_api, but should provide a much smaller and more manageable introduction into how to write a py2wasm project. I'll update this OP as needed with our resource list and making goals more defined.

How to currently run this

  1. python -m venv .venv
  2. source .venv/bin/activate
  3. pip install -r requirements.txt
  4. python app.py (a local development server)
  5. open http://127.0.0.1:5000

What do I mean by "make into a WASM project"

WASM (WebAssembly) is a standard where a program is compiled into bytecode, which is then compiled to a browser host's native code and run in sort of a virtual machine, in the browser. There are APIs exposed to allow a WebAssembly program to 'do stuff' on the page, interact with files and attached devices. Most programs written in C/C++ can be easily ported via the Emscripten WASM compiler. I have a small amount of experience with C/C++ but have a lot of experience with python. There are many python compilers that will compile python programs into C. py2wasm will take a python program, compile it to C, then will be compiled into WebAssembly, then when downloaded by a visiting client to a website, be compiled into the architecture of the client platform.

It's rather slow for the user because first it has to download, then compile, then load. There are techniques to make this less annoying for the user, such as loading some content that they need to read/interact with while the WASM is compiling in the background. There's also the consideration that JS wants to work with 'promises' for async - and I don't know how this will work for py2wasm.

That's why I think this project would be good to experiment with, because adding/changing functionality is easy on account of the small size of the program.

Resources

  1. The WASM compiler (exposes WASM API): https://emscripten.org/
    0.1 "wasmer" WASM runtime ?? https://github.com/wasmerio/wasmer
  2. The python wasm compiler
    1.1. https://wasmer.io/posts/py2wasm-a-python-to-wasm-compiler
    1.2. https://github.com/wasmerio/py2wasm

Goals

  1. Make a working WASM module from a python program (py2wasm) that just shows an introduction web page with an upload area
  2. Implement the file date checker/git repo commit checker.
  3. Implement the import checker (get imports that are not part of stdlib)
  4. Implement the pypi.org version page parser

Secondary Goals

These are for researching features that are desirable in kjv_api_v2 and not needfully integrated into this project. Since this project is pretty simple, using it as a sort of template for

  1. Database management
    0.1 Distributed PostgreSQL (discuss after individual research and mutual understanding of needs for kjv_apiv2 for appropriate type but Multi-Master Asynchronous Replication style might be appropriate)
    0.2 Job queuing system for handling volume of no. 3 below

  2. key/user/group management (for subscribers)
    1.1. Just general experimentation with the best way to manage users and groups. kjv_api_v2 will have to deal with these and there needs to be something easy for the user and easy for management.
    1.2. Payment processing integration with automatic assignment of user keys and group IDs, locking certain features behind the presence of the ID.
    1.3. Key sharing detection and prevention (max-device-per-key limit)

  3. Historical records management for subscribers
    2.1. Local searches and analysis sent to control server for statistics and user ownership retrieval.

  4. Forward files to server for remote processing.
    3.1. Premium feature for subscribers to allow for faster/better quality processing.
    3.2. Expensive operation!

For future reference for kjv_api_v2 (add links as needed)

https://developer.chrome.com/blog/io24-webassembly-webgpu-1

cc @pyrate This is another project of mine I just 'finished', not directly related to the v2 of the kjv_api, but should provide a much smaller and more manageable introduction into how to write a py2wasm project. I'll update this OP as needed with our resource list and making goals more defined. ## How to currently run this 1. `python -m venv .venv` 2. `source .venv/bin/activate` 3. `pip install -r requirements.txt` 4. `python app.py` (a local development server) 5. open `http://127.0.0.1:5000` ## What do I mean by "make into a WASM project" WASM (WebAssembly) is a standard where a program is compiled into bytecode, which is then compiled to a browser host's native code and run in sort of a virtual machine, in the browser. There are APIs exposed to allow a WebAssembly program to 'do stuff' on the page, interact with files and attached devices. Most programs written in C/C++ can be easily ported via the Emscripten WASM compiler. I have a small amount of experience with C/C++ but have a lot of experience with python. There are many python compilers that will compile python programs into C. py2wasm will take a python program, compile it to C, then will be compiled into WebAssembly, then when downloaded by a visiting client to a website, be compiled into the architecture of the client platform. It's rather slow for the user because first it has to download, then compile, then load. There are techniques to make this less annoying for the user, such as loading some content that they need to read/interact with while the WASM is compiling in the background. There's also the consideration that JS wants to work with 'promises' for async - and I don't know how this will work for py2wasm. That's why I think this project would be good to experiment with, because adding/changing functionality is easy on account of the small size of the program. ### Resources 0. The WASM compiler (exposes WASM API): https://emscripten.org/ 0.1 "wasmer" WASM runtime ?? https://github.com/wasmerio/wasmer 1. The python wasm compiler 1.1. https://wasmer.io/posts/py2wasm-a-python-to-wasm-compiler 1.2. https://github.com/wasmerio/py2wasm ## Goals 1. Make a working WASM module from a python program (py2wasm) that just shows an introduction web page with an upload area 2. Implement the file date checker/git repo commit checker. 3. Implement the import checker (get imports that are not part of stdlib) 4. Implement the pypi.org version page parser ## Secondary Goals These are for researching features that are desirable in kjv_api_v2 and not needfully integrated into this project. Since this project is pretty simple, using it as a sort of template for 0. Database management 0.1 Distributed PostgreSQL (discuss after individual research and mutual understanding of needs for kjv_apiv2 for appropriate type but Multi-Master Asynchronous Replication style might be appropriate) 0.2 Job queuing system for handling volume of no. 3 below 1. key/user/group management (for subscribers) 1.1. Just general experimentation with the best way to manage users and groups. kjv_api_v2 will have to deal with these and there needs to be something easy for the user and easy for management. 1.2. Payment processing integration with automatic assignment of user keys and group IDs, locking certain features behind the presence of the ID. 1.3. Key sharing detection and prevention (max-device-per-key limit) 2. Historical records management for subscribers 2.1. Local searches and analysis sent to control server for statistics and user ownership retrieval. 3. Forward files to server for remote processing. 3.1. Premium feature for subscribers to allow for faster/better quality processing. 3.2. Expensive operation! ### For future reference for kjv_api_v2 (add links as needed) https://developer.chrome.com/blog/io24-webassembly-webgpu-1
tyler added this to the Convert to WASM Project project 2024-12-26 00:42:11 +00:00
tyler commented 2024-12-28 15:03:46 +00:00
Author
Owner
Copy Link

I did not expect just getting a right build environment to be this much of a pain in the ass.

Struggled to get pyenv to work right on my system but here's the setup that worked for me:

  1. Clone this repo
  2. curl https://pyenv.run | bash
  3. It's going to say you need to modify your .bashrc or .bash_profile (this is what I put in ~/.bashrc)
export PYENV_ROOT="$HOME/.pyenv"
[[ -d $PYENV_ROOT/bin ]] && export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init - bash)"
eval "$(pyenv virtualenv-init -)"
  1. exec "$SHELL"
  2. pyenv install 3.11
  3. In the repo folder... pyenv virtualenv .venv
    6.1 (pyenv local 3.11?????? - idk how it got set to this, I had trouble with this part, it wouldn't work reliably)
  4. source activate .venv
  5. pip install -r requirements.txt
    8.1 pip install flask
    8.2 pip install py2wasm
    Then:
  6. py2wasm app.py

This will compile app.py, and it takes several minutes. But it requires that you be in python3.11 with pyenv, activated the .venv, and it should just werk ™️

$ py2wasm app.py 
py2wasm: wasi-sdk not found in /home/tyler/.pyenv/versions/.venv/lib/python3.11/site-packages/nuitka/wasi-sdk/21/sdk-Linux, downloading the SDK
py2wasm: Downloading wasi-sdk-21.0-linux.tar.gz
wasi-sdk-21.0-linux.tar.gz: 100%|███████████████████████████████████| 74.9M/74.9M [00:00<00:00, 99.0MMiB/s]
py2wasm: Extracting wasi-sdk-21.0-linux.tar.gz
py2wasm: wasi-sdk installed at: /home/tyler/.pyenv/versions/.venv/lib/python3.11/site-packages/nuitka/wasi-sdk/21/sdk
Nuitka-Options: Used command line options: app.py --standalone --static-libpython=yes --disable-ccache
Nuitka-Options: --lto=yes
Nuitka-Options: --output-dir=/home/tyler/.pyenv/versions/.venv/lib/python3.11/site-packages/nuitka/__py2wasm
Nuitka-Options: --output-filename=output.wasm --remove-output
Nuitka: Starting Python compilation with Nuitka '2.6.3' on Python '3.11' commercial grade 'not installed'.
Nuitka-Plugins:anti-bloat: Not including '_bisect' automatically in order to avoid bloat, but this may
Nuitka-Plugins:anti-bloat: cause: may slow down by using fallback implementation.
Nuitka-Plugins:anti-bloat: Not including '_json' automatically in order to avoid bloat, but this may cause:
Nuitka-Plugins:anti-bloat: may slow down by using fallback implementation.
Nuitka: Completed Python level compilation and optimization.
Nuitka: Generating source code for C backend compiler.
Nuitka: Running data composer tool for optimal constant value handling.                             
Nuitka: Running C compilation via Scons.
Nuitka-Scons: Backend C compiler:
Nuitka-Scons: ~/.pyenv/versions/.venv/lib/python3.11/site-packages/nuitka/wasi-sdk/21/sdk-Linux/bin/clang
Nuitka-Scons: (clang 17.0.6).
Nuitka-Scons: Backend linking program with 289 files (no progress information available for this stage).
wasm-ld: warning: unknown -z value: noexecstack
Nuitka-Plugins:data-files: Included data file 'certifi/cacert.pem' due to package data for 'certifi'.
Nuitka: Removing build directory                                                            
Nuitka: '/home/tyler/.pyenv/versions/.venv/lib/python3.11/site-packages/nuitka/__py2wasm/app.build'.
Nuitka: Successfully created
Nuitka: '/home/tyler/.pyenv/versions/.venv/lib/python3.11/site-packages/nuitka/__py2wasm/app.dist/output.wasm'.
py2wasm: File created successfully: app.wasm

This produces app.wasm, a 64 megabyte file. I am certain there are ways to trim this down, but next have to figure out how to serve it (to see if it even works: I have no idea what will happen lmao)

Looks like: https://developer.mozilla.org/en-US/docs/WebAssembly/Loading_and_running (I need to read)

Also: https://github.com/wasmerio/wasmer looks like a deskop/serverside runtime??

Running app.wasm with wasmer does not do anything, but that might make sense because no methods are exposed to webassembly that way.

I did not expect just getting a right build environment to be this much of a pain in the ass. Struggled to get pyenv to work right on my system but here's the setup that worked for me: 1. Clone this repo 2. `curl https://pyenv.run | bash` 3. It's going to say you need to modify your .bashrc or .bash_profile (this is what I put in ~/.bashrc) ``` export PYENV_ROOT="$HOME/.pyenv" [[ -d $PYENV_ROOT/bin ]] && export PATH="$PYENV_ROOT/bin:$PATH" eval "$(pyenv init - bash)" eval "$(pyenv virtualenv-init -)" ``` 4. `exec "$SHELL"` 5. `pyenv install 3.11` 6. In the repo folder... `pyenv virtualenv .venv` 6.1 (`pyenv local 3.11`?????? - idk how it got set to this, I had trouble with this part, it wouldn't work reliably) 7. `source activate .venv` 8. `pip install -r requirements.txt` 8.1 `pip install flask` 8.2 `pip install py2wasm` Then: 9. `py2wasm app.py` This will compile app.py, and it takes *several* minutes. But it requires that you be in python3.11 with pyenv, activated the .venv, and it should just werk ™️ ``` $ py2wasm app.py py2wasm: wasi-sdk not found in /home/tyler/.pyenv/versions/.venv/lib/python3.11/site-packages/nuitka/wasi-sdk/21/sdk-Linux, downloading the SDK py2wasm: Downloading wasi-sdk-21.0-linux.tar.gz wasi-sdk-21.0-linux.tar.gz: 100%|███████████████████████████████████| 74.9M/74.9M [00:00<00:00, 99.0MMiB/s] py2wasm: Extracting wasi-sdk-21.0-linux.tar.gz py2wasm: wasi-sdk installed at: /home/tyler/.pyenv/versions/.venv/lib/python3.11/site-packages/nuitka/wasi-sdk/21/sdk Nuitka-Options: Used command line options: app.py --standalone --static-libpython=yes --disable-ccache Nuitka-Options: --lto=yes Nuitka-Options: --output-dir=/home/tyler/.pyenv/versions/.venv/lib/python3.11/site-packages/nuitka/__py2wasm Nuitka-Options: --output-filename=output.wasm --remove-output Nuitka: Starting Python compilation with Nuitka '2.6.3' on Python '3.11' commercial grade 'not installed'. Nuitka-Plugins:anti-bloat: Not including '_bisect' automatically in order to avoid bloat, but this may Nuitka-Plugins:anti-bloat: cause: may slow down by using fallback implementation. Nuitka-Plugins:anti-bloat: Not including '_json' automatically in order to avoid bloat, but this may cause: Nuitka-Plugins:anti-bloat: may slow down by using fallback implementation. Nuitka: Completed Python level compilation and optimization. Nuitka: Generating source code for C backend compiler. Nuitka: Running data composer tool for optimal constant value handling. Nuitka: Running C compilation via Scons. Nuitka-Scons: Backend C compiler: Nuitka-Scons: ~/.pyenv/versions/.venv/lib/python3.11/site-packages/nuitka/wasi-sdk/21/sdk-Linux/bin/clang Nuitka-Scons: (clang 17.0.6). Nuitka-Scons: Backend linking program with 289 files (no progress information available for this stage). wasm-ld: warning: unknown -z value: noexecstack Nuitka-Plugins:data-files: Included data file 'certifi/cacert.pem' due to package data for 'certifi'. Nuitka: Removing build directory Nuitka: '/home/tyler/.pyenv/versions/.venv/lib/python3.11/site-packages/nuitka/__py2wasm/app.build'. Nuitka: Successfully created Nuitka: '/home/tyler/.pyenv/versions/.venv/lib/python3.11/site-packages/nuitka/__py2wasm/app.dist/output.wasm'. py2wasm: File created successfully: app.wasm ``` This produces app.wasm, a 64 megabyte file. I am certain there are ways to trim this down, but next have to figure out how to serve it (to see if it even works: I have no idea what will happen lmao) Looks like: https://developer.mozilla.org/en-US/docs/WebAssembly/Loading_and_running (I need to read) Also: https://github.com/wasmerio/wasmer looks like a deskop/serverside runtime?? Running app.wasm with wasmer does not do anything, but that might make sense because no methods are exposed to webassembly that way.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project Convert to WASM Project
Assignees
Clear assignees
pyrate tyler
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: tyler/python-requirements-txt-generator#1
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?