Welcome to Crawler’s documentation!

Crawler is an open source web crawler for Swiftea. It can’t be run by contributors because it needs private_data.py which is not upload for obvious reasons.

• 1. Modules
  • 1.1. crawler main module
  • 1.2. statistiques module
• 2. Package
  • 2.1. swiftea_bot.module module
  • 2.2. swiftea_bot.data module
  • 2.3. swiftea_bot.file_manager module
  • 2.4. crawling.web_connection module
  • 2.5. crawling.connection module
  • 2.6. crawling.site_informations module
  • 2.7. crawling.searches module
  • 2.8. crawling.parsers module
  • 2.9. database.database module
  • 2.10. database.database_manager module
  • 2.11. database.database_swiftea module
  • 2.12. index.index module
  • 2.13. index.inverted_index module
  • 2.14. index.ftp_manager module
  • 2.15. index.ftp_swiftea module
• 3. Tests
  • 3.1. tests.run_tests
  • 3.2. tests.swiftea_bot_test
  • 3.3. tests.crawling_test
  • 3.4. tests.database_test
  • 3.5. tests.index_test
  • 3.6. tests.crawler_test
  • 3.7. tests.global_test
  • 3.8. tests.test_data

Indices and tables

• Index
• Module Index
• Search Page