Python HTML CSS JavaScript Other
Latest commit 052a038 Dec 2, 2017 @JodanJodan JodanJodan committed with TheAMM Add tabindexes to login elements (#420)
Fixes issue with password managers (e.g. KeePass) tabbing to 'Forgot your password?' link instead of password field.
Failed to load latest commit information.
.github Words (#355) Sep 1, 2017
configs moved some files May 17, 2017
migrations [Schema+config change] Comment editing (#396) Nov 5, 2017
nyaa Add tabindexes to login elements (#420) Dec 2, 2017
tests Refactor into an app factory [2 of 2] (#322) Aug 1, 2017
torrent_cache Initial commit. May 12, 2017
utils Fix lint issues in utils/ Aug 5, 2017
.gitattributes Add .gitattributes (#310) Jul 26, 2017
.gitignore Fix RFC822 filters + More tests (#257) Jul 7, 2017
.travis.yml Nyaa development helper (tool) (#324) Aug 5, 2017
LICENSE Add license (GPLv3) May 12, 2017 Nyaa development helper (tool) (#324) Aug 5, 2017 Refactor into an app factory [2 of 2] (#322) Aug 1, 2017 Email blacklist (#419) Nov 23, 2017 hooked up ES... 90% done, need to figure out how to generate magnet URIs May 16, 2017 Refactor into an app factory [2 of 2] (#322) Aug 1, 2017 Refactor into an app factory [2 of 2] (#322) Aug 1, 2017 Nyaa development helper (tool) (#324) Aug 5, 2017
es_mapping.yml [ES Schema change] Multi-field search display_name to match words ove… Jun 5, 2017
es_sync_config.example.json Update to index both torrent flavors, rename sync_es … May 27, 2017 fix broken Sep 17, 2017 Nyaa development helper (tool) (#324) Aug 5, 2017
requirements.txt Update email verification, add Mailgun backend (#380) Oct 8, 2017 Refactor into an app factory [2 of 2] (#322) Aug 1, 2017
setup.cfg Nyaa development helper (tool) (#324) Aug 5, 2017 Fix permissions May 29, 2017
trackers.txt Remove tracker limit and always add our trackers (#417) Nov 22, 2017
uwsgi.ini Initial commit. May 12, 2017

NyaaV2 Build Status

Setting up for development

This project uses Python 3.6. There are features used that do not exist in 3.5, so make sure to use Python 3.6.
This guide also assumes you 1) are using Linux and 2) are somewhat capable with the commandline.
It's not impossible to run Nyaa on Windows, but this guide doesn't focus on that.

Code Quality:

  • Before we get any deeper, remember to follow PEP8 style guidelines and run ./ lint before committing to see a list of warnings/problems.
    • You may also use ./ fix && ./ isort to automatically fix some of the issues reported by the previous command.
  • Other than PEP8, try to keep your code clean and easy to understand, as well. It's only polite!

Running Tests

The tests folder contains tests for the the nyaa module and the webserver. To run the tests:

  • Make sure that you are in the python virtual environment.
  • Run ./ test while in the repository directory.

Setting up Pyenv

pyenv eases the use of different Python versions, and as not all Linux distros offer 3.6 packages, it's right up our alley.

Setting up MySQL/MariaDB database

You may use SQLite but the current support for it in this project is outdated and rather unsupported.

  • Enable USE_MYSQL flag in
  • Install latest mariadb by following instructions here
    • Tested versions: mysql Ver 15.1 Distrib 10.0.30-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2
  • Run the following commands logged in as your root db user (substitute for your own values if desired):
    • CREATE USER 'test'@'localhost' IDENTIFIED BY 'test123';
    • GRANT ALL PRIVILEGES ON *.* TO 'test'@'localhost';

Finishing up

  • Run python to create the database and import categories
    • Follow the advice of and run ./ stamp head to mark the database version for Alembic
  • Start the dev server with python
  • When you are finished developing, deactivate your virtualenv with pyenv deactivate or source deactivate (or just close your shell session)

You're now ready for simple testing and development!
Continue below to learn about database migrations and enabling the advanced search engine, Elasticsearch.

Database migrations

  • Database migrations are done with flask-Migrate, a wrapper around Alembic.
  • If someone has made changes in the database schema and included a new migration script:
    • If your database has never been marked by Alembic (you're on a database from before the migrations), run ./ stamp head before pulling the new migration script(s).
      • If you already have the new scripts, check the output of ./ history instead and choose a hash that matches your current database state, then run ./ stamp <hash>.
    • Update your branch (eg. git fetch && git rebase origin/master)
    • Run ./ upgrade head to run the migration. Done!
  • If you have made a change in the database schema:
    • Save your changes in and ensure the database schema matches the previous version (ie. your new tables/columns are not added to the live database)
    • Run ./ migrate -m "Short description of changes" to automatically generate a migration script for the changes
      • Check the script (migrations/versions/...) and make sure it works! Alembic may not able to notice all changes.
    • Run ./ upgrade to run the migration and verify the upgrade works.
      • (Run ./ downgrade to verify the downgrade works as well, then upgrade again)

Setting up and enabling Elasticsearch

Installing Elasticsearch

Setting up ES

  • Run ./ to create the indices for the torrents: nyaa and sukebei
    • The output should show acknowledged: true twice
  • Stop the Nyaa app if you haven't already
  • Run python to import all the torrents (on nyaa and sukebei) into the ES indices.
    • This may take some time to run if you have plenty of torrents in your database.

Enable the USE_ELASTIC_SEARCH flag in and (re)start the application.
Elasticsearch should now be functional! The ES indices won't be updated "live" with the current setup, continue below for instructions on how to hook Elasticsearch up to MySQL binlog.

However, take note that binglog is not necessary for simple ES testing and development; you can simply run from time to time to reindex all the torrents.

Enabling MySQL Binlogging

  • Edit your MariaDB/MySQL server configuration and add the following under [mariadb]:
  • Restart MariaDB/MySQL (sudo service mysql restart)
  • Copy the example configuration (es_sync_config.example.json) as es_sync_config.json and adjust options in it to your liking (verify the connection options!)
  • Connect to mysql as root
    • Verify that the result of SHOW VARIABLES LIKE 'binlog_format'; is ROW
    • Execute GRANT REPLICATION SLAVE ON *.* TO 'username'@'localhost'; to allow your configured user access to the binlog

Setting up keeps the Elasticsearch indices updated by reading the binlog and pushing the changes to the ES indices.

  • Make sure es_sync_config.json is configured with the user you grated the REPLICATION permissions
  • Run and copy the outputted JSON into the file specified by save_loc in your es_sync_config.json
  • Run as-is or, for actual deployment, set it up as a service and run it, preferably as the system/root
    • Make sure runs within the venv with the right dependencies!

You're done! The script should now be feeding updates from the database to Elasticsearch.
Take note, however, that the specified ES index refresh interval is 30 seconds, which may feel like a long time on local development. Feel free to adjust it or poke Elasticsearch yourself!