Commit 59f453f0 authored by Xavier Barbosa's avatar Xavier Barbosa

write some documentation

parent 19f27fb1
Pipeline #522 passed with stages
......@@ -3,6 +3,7 @@
stages:
- build
- test
- release
Prepare container:
stage: build
......@@ -20,3 +21,15 @@ Test:
- py.test tests/ --cov aiodisque --cov-report term-missing --flake8
tags:
- python3.5
Release to pypi:
stage: release
image: errorist/aiodisque:latest
script:
- python setup.py sdist bdist_wheel
- twine upload -u $PYPI_USER -p $PYPI_PASSWORD dist/*
tags:
- python3.5
only:
- /^v[\d\.]+.*$/
allow_failure: true
......@@ -20,7 +20,7 @@ RUN cd disque-${DISQUE_VERSION} \
&& rm -rf disque-${DISQUE_VERSION}
RUN chmod +x /usr/local/bin/disque
RUN chmod +x /usr/local/bin/disque-server
RUN python -m pip install hiredis pytest pytest-cov pytest-flake8
RUN python -m pip install hiredis pytest pytest-cov pytest-flake8 twine wheel
EXPOSE $DISQUE_PORT
......
AIO Disque
==========
Python3.5 & Asyncio client for disque_ message broker.
Python3.5 & Asyncio client for Disque_ message broker.
Installation
......@@ -23,6 +23,7 @@ Usage::
client = Disque()
job_id = await client.sendjob('queue', 'body')
API Reference
-------------
......@@ -30,7 +31,7 @@ The `official Disque command documentation`_ does a great job of explaining
each command in detail. There are a few exceptions:
* each method are lowered
* async keywords are replaced by asap
* ``async`` keywords are replaced by asynchronous
In addition to the changes above, it implements some async sugar:
......@@ -45,7 +46,7 @@ In addition to the changes above, it implements some async sugar:
async for job in client.jscan_iter(count=128):
print(job)
* mimic an asyncio Queue::
* There is also an experimentaton that try to mimic an asyncio.Queue::
from aiodisque.queue import Queue
queue = JobsQueue('queue', client)
......@@ -53,5 +54,5 @@ In addition to the changes above, it implements some async sugar:
job = await queue.get()
assert job.id == job_id
.. _disque: https://github.com/antirez/disque
.. _Disque: https://github.com/antirez/disque
.. _`official Disque command documentation`: https://github.com/antirez/disque#main-api
from .client import *
from .connections import *
from .iterators import *
from .queues import *
from .scanners import *
__all__ = client.__all__ + connections.__all__ + queues.__all__
__all__ = (client.__all__ +
connections.__all__ +
iterators.__all__ +
queues.__all__ +
scanners.__all__)
from ._version import get_versions
__version__ = get_versions()['version']
......
This diff is collapsed.
from collections.abc import AsyncIterator
__all__ = ['JobsIterator']
class JobsIterator(AsyncIterator):
......
......@@ -7,13 +7,13 @@ class JobsQueue:
class Empty(Exception):
"""
Exception raised when non-blocking :meth:`~EventsQueue.get`
Exception raised when non-blocking :meth:`~JobsQueue.get`
is called on a :class:`JobsQueue` object which is empty.
"""
class Full(Exception):
"""
Exception raised when :meth:`~EventsQueue.put` is called on a
Exception raised when :meth:`~JobsQueue.put` is called on a
:class:`JobsQueue` object which is full.
"""
......@@ -78,7 +78,8 @@ class JobsQueue:
job = getattr(job, 'body', job)
response = await self.client.addjob(self.name, job, ms_timeout=0,
replicate=None, delay=None,
retry=None, ttl=None, asap=False,
retry=None, ttl=None,
asynchronous=False,
maxlen=self.maxsize or None)
return response
......
from collections import deque
from collections.abc import AsyncIterator
__all__ = ['JobsScanner', 'QueuesScanner']
class ScannerIterator(AsyncIterator):
......@@ -37,6 +39,18 @@ class JobsScanner(ScannerIterator):
def __init__(self, client, *,
states=None, count=None, queue=None, reply=None):
"""Iter thru jobs.
Parameters:
client (Disque): disque client
count (int): An hit about how much work to do per iteration
queue (str): Return only jobs in the specified queue
states (str): Return jobs in the specified state. Can be used
multiple times for a logic OR
reply (str): Job reply type. Type can be all or id. Default is to
report just the job ID. If all is specified the full
job state is returned like for the SHOW command
"""
func = client.jscan
func_args = states or []
func_kwargs = {
......@@ -51,6 +65,18 @@ class QueuesScanner(ScannerIterator):
def __init__(self, client, *,
count=None, minlen=None, maxlen=None, import_rate=None):
"""Iter thru queues.
Parameters:
client (Disque): disque client
count (int): An hit about how much work to do per iteration
minlen (int): Don't return elements with less than
count jobs queued
maxlen (int): Don't return elements with more than
count jobs queued
import_rate <rate>: Only return elements with an job import rate
(from other nodes) >= rate
"""
func = client.qscan
func_args = []
func_kwargs = {
......
This diff is collapsed.
# Makefile for Sphinx documentation
#
# You can set these variables from the command line.
SPHINXOPTS =
SPHINXBUILD = sphinx-build
PAPER =
BUILDDIR = _build
# User-friendly check for sphinx-build
ifeq ($(shell which $(SPHINXBUILD) >/dev/null 2>&1; echo $$?), 1)
$(error The '$(SPHINXBUILD)' command was not found. Make sure you have Sphinx installed, then set the SPHINXBUILD environment variable to point to the full path of the '$(SPHINXBUILD)' executable. Alternatively you can add the directory with the executable to your PATH. If you don't have Sphinx installed, grab it from http://sphinx-doc.org/)
endif
# Internal variables.
PAPEROPT_a4 = -D latex_paper_size=a4
PAPEROPT_letter = -D latex_paper_size=letter
ALLSPHINXOPTS = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) .
# the i18n builder cannot share the environment and doctrees with the others
I18NSPHINXOPTS = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) .
.PHONY: help clean html dirhtml singlehtml pickle json htmlhelp qthelp devhelp epub latex latexpdf text man changes linkcheck doctest coverage gettext
help:
@echo "Please use \`make <target>' where <target> is one of"
@echo " html to make standalone HTML files"
@echo " dirhtml to make HTML files named index.html in directories"
@echo " singlehtml to make a single large HTML file"
@echo " pickle to make pickle files"
@echo " json to make JSON files"
@echo " htmlhelp to make HTML files and a HTML help project"
@echo " qthelp to make HTML files and a qthelp project"
@echo " applehelp to make an Apple Help Book"
@echo " devhelp to make HTML files and a Devhelp project"
@echo " epub to make an epub"
@echo " latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter"
@echo " latexpdf to make LaTeX files and run them through pdflatex"
@echo " latexpdfja to make LaTeX files and run them through platex/dvipdfmx"
@echo " text to make text files"
@echo " man to make manual pages"
@echo " texinfo to make Texinfo files"
@echo " info to make Texinfo files and run them through makeinfo"
@echo " gettext to make PO message catalogs"
@echo " changes to make an overview of all changed/added/deprecated items"
@echo " xml to make Docutils-native XML files"
@echo " pseudoxml to make pseudoxml-XML files for display purposes"
@echo " linkcheck to check all external links for integrity"
@echo " doctest to run all doctests embedded in the documentation (if enabled)"
@echo " coverage to run coverage check of the documentation (if enabled)"
clean:
rm -rf $(BUILDDIR)/*
html:
$(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html
@echo
@echo "Build finished. The HTML pages are in $(BUILDDIR)/html."
dirhtml:
$(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml
@echo
@echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml."
singlehtml:
$(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml
@echo
@echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml."
pickle:
$(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle
@echo
@echo "Build finished; now you can process the pickle files."
json:
$(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json
@echo
@echo "Build finished; now you can process the JSON files."
htmlhelp:
$(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp
@echo
@echo "Build finished; now you can run HTML Help Workshop with the" \
".hhp project file in $(BUILDDIR)/htmlhelp."
qthelp:
$(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp
@echo
@echo "Build finished; now you can run "qcollectiongenerator" with the" \
".qhcp project file in $(BUILDDIR)/qthelp, like this:"
@echo "# qcollectiongenerator $(BUILDDIR)/qthelp/AIODisque.qhcp"
@echo "To view the help file:"
@echo "# assistant -collectionFile $(BUILDDIR)/qthelp/AIODisque.qhc"
applehelp:
$(SPHINXBUILD) -b applehelp $(ALLSPHINXOPTS) $(BUILDDIR)/applehelp
@echo
@echo "Build finished. The help book is in $(BUILDDIR)/applehelp."
@echo "N.B. You won't be able to view it unless you put it in" \
"~/Library/Documentation/Help or install it in your application" \
"bundle."
devhelp:
$(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp
@echo
@echo "Build finished."
@echo "To view the help file:"
@echo "# mkdir -p $$HOME/.local/share/devhelp/AIODisque"
@echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/AIODisque"
@echo "# devhelp"
epub:
$(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub
@echo
@echo "Build finished. The epub file is in $(BUILDDIR)/epub."
latex:
$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
@echo
@echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex."
@echo "Run \`make' in that directory to run these through (pdf)latex" \
"(use \`make latexpdf' here to do that automatically)."
latexpdf:
$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
@echo "Running LaTeX files through pdflatex..."
$(MAKE) -C $(BUILDDIR)/latex all-pdf
@echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex."
latexpdfja:
$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
@echo "Running LaTeX files through platex and dvipdfmx..."
$(MAKE) -C $(BUILDDIR)/latex all-pdf-ja
@echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex."
text:
$(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text
@echo
@echo "Build finished. The text files are in $(BUILDDIR)/text."
man:
$(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man
@echo
@echo "Build finished. The manual pages are in $(BUILDDIR)/man."
texinfo:
$(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo
@echo
@echo "Build finished. The Texinfo files are in $(BUILDDIR)/texinfo."
@echo "Run \`make' in that directory to run these through makeinfo" \
"(use \`make info' here to do that automatically)."
info:
$(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo
@echo "Running Texinfo files through makeinfo..."
make -C $(BUILDDIR)/texinfo info
@echo "makeinfo finished; the Info files are in $(BUILDDIR)/texinfo."
gettext:
$(SPHINXBUILD) -b gettext $(I18NSPHINXOPTS) $(BUILDDIR)/locale
@echo
@echo "Build finished. The message catalogs are in $(BUILDDIR)/locale."
changes:
$(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes
@echo
@echo "The overview file is in $(BUILDDIR)/changes."
linkcheck:
$(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck
@echo
@echo "Link check complete; look for any errors in the above output " \
"or in $(BUILDDIR)/linkcheck/output.txt."
doctest:
$(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest
@echo "Testing of doctests in the sources finished, look at the " \
"results in $(BUILDDIR)/doctest/output.txt."
coverage:
$(SPHINXBUILD) -b coverage $(ALLSPHINXOPTS) $(BUILDDIR)/coverage
@echo "Testing of coverage in the sources finished, look at the " \
"results in $(BUILDDIR)/coverage/python.txt."
xml:
$(SPHINXBUILD) -b xml $(ALLSPHINXOPTS) $(BUILDDIR)/xml
@echo
@echo "Build finished. The XML files are in $(BUILDDIR)/xml."
pseudoxml:
$(SPHINXBUILD) -b pseudoxml $(ALLSPHINXOPTS) $(BUILDDIR)/pseudoxml
@echo
@echo "Build finished. The pseudo-XML files are in $(BUILDDIR)/pseudoxml."
"""
pyspecific.py
~~~~~~~~~~~~~
Sphinx extension with Python doc-specific markup.
:copyright: 2008-2014 by Georg Brandl.
:license: Python license.
"""
import inspect
from sphinx import addnodes
from sphinx.domains.python import PyModulelevel, PyClassmember
from sphinx.ext.autodoc import FunctionDocumenter, MethodDocumenter
class PyAsyncMixin(object):
def handle_signature(self, sig, signode):
ret = super().handle_signature(sig, signode)
signode.insert(0, addnodes.desc_annotation('async ', 'async '))
return ret
class PyAsyncFunction(PyAsyncMixin, PyModulelevel):
def run(self):
self.name = 'py:function'
return PyModulelevel.run(self)
class PyAsyncMethod(PyAsyncMixin, PyClassmember):
def run(self):
self.name = 'py:method'
return PyClassmember.run(self)
class PyGeneratorMixin(object):
def handle_signature(self, sig, signode):
ret = super().handle_signature(sig, signode)
signode.insert(0, addnodes.desc_annotation('generator ', 'generator '))
return ret
class PyGeneratorFunction(PyGeneratorMixin, PyModulelevel):
def run(self):
self.name = 'py:function'
return PyModulelevel.run(self)
class PyGeneratorMethod(PyGeneratorMixin, PyClassmember):
def run(self):
self.name = 'py:method'
return PyClassmember.run(self)
class PyCoroutineMixin(object):
def handle_signature(self, sig, signode):
ret = super().handle_signature(sig, signode)
signode.insert(0, addnodes.desc_annotation('coroutine ', 'coroutine '))
return ret
class PyCoroutineFunction(PyCoroutineMixin, PyModulelevel):
def run(self):
self.name = 'py:function'
return PyModulelevel.run(self)
class PyCoroutineMethod(PyCoroutineMixin, PyClassmember):
def run(self):
self.name = 'py:method'
return PyClassmember.run(self)
class PyTaskMixin(object):
def handle_signature(self, sig, signode):
ret = super().handle_signature(sig, signode)
signode.insert(0, addnodes.desc_annotation('task ', 'task '))
return ret
class PyTaskFunction(PyTaskMixin, PyModulelevel):
def run(self):
self.name = 'py:function'
return PyModulelevel.run(self)
class PyTaskMethod(PyTaskMixin, PyClassmember):
def run(self):
self.name = 'py:method'
return PyClassmember.run(self)
class FunctionDocumenter(FunctionDocumenter):
"""
Specialized Documenter subclass for coroutines.
"""
def import_object(self):
ret = super().import_object()
if self.directivetype in ('task',
'async',
'coroutine',
'generator'):
return ret
obj = self.parent.__dict__.get(self.object_name)
if getattr(obj, '_is_task', False):
self.directivetype = 'task'
self.member_order = self.member_order + 2
elif inspect.isawaitable(obj):
self.directivetype = 'async'
self.member_order = self.member_order + 2
elif getattr(obj, '_is_coroutine', False):
self.directivetype = 'coroutine'
self.member_order = self.member_order - 2
elif inspect.isgeneratorfunction(obj):
self.directivetype = 'generator'
self.member_order = self.member_order + 3
return ret
class MethodDocumenter(MethodDocumenter):
"""
Specialized Documenter subclass for coroutines methods.
"""
def import_object(self):
ret = super().import_object()
if self.directivetype in ('taskmethod',
'asyncmethod',
'coroutinemethod',
'generatormethod'):
return ret
obj = self.parent.__dict__.get(self.object_name)
if getattr(obj, '_is_task', False):
self.directivetype = 'taskmethod'
self.member_order = self.member_order - 2
elif inspect.isawaitable(obj):
self.directivetype = 'asyncmethod'
self.member_order = self.member_order + 2
elif getattr(obj, '_is_coroutine', False):
self.directivetype = 'coroutinemethod'
self.member_order = self.member_order - 2
elif inspect.isgeneratorfunction(obj):
self.directivetype = 'generatormethod'
self.member_order = self.member_order + 3
return ret
def setup(app):
app.add_directive_to_domain('py', 'async', PyAsyncFunction)
app.add_directive_to_domain('py', 'asyncmethod', PyAsyncMethod)
app.add_directive_to_domain('py', 'generator', PyGeneratorFunction)
app.add_directive_to_domain('py', 'generatormethod', PyGeneratorMethod)
app.add_directive_to_domain('py', 'coroutine', PyCoroutineFunction)
app.add_directive_to_domain('py', 'coroutinemethod', PyCoroutineMethod)
app.add_directive_to_domain('py', 'task', PyTaskFunction)
app.add_directive_to_domain('py', 'taskmethod', PyTaskMethod)
app.add_autodocumenter(FunctionDocumenter)
app.add_autodocumenter(MethodDocumenter)
.. py:module:: aiodisque
.. _api:
API
===
.. autoclass:: Disque
:members:
:undoc-members:
.. autoclass:: Job
:members:
:undoc-members:
.. autoclass:: Cursor
:members:
:undoc-members:
.. autoclass:: JobsIterator
:members:
:undoc-members:
.. autoclass:: JobsQueue
:members:
:undoc-members:
.. autoclass:: QueuesScanner
:members:
:undoc-members:
.. autoclass:: JobsScanner
:members:
:undoc-members:
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
#
# AIO Disque documentation build configuration file, created by
# sphinx-quickstart on Thu Jan 7 12:14:19 2016.
#
# This file is execfile()d with the current directory set to its
# containing dir.
#
# Note that not all possible configuration values are present in this
# autogenerated file.
#
# All configuration values have a default; values that are commented out
# serve to show the default.
import sys
import os
import shlex
sys.path.insert(0, os.path.abspath('..'))
sys.path.append(os.path.abspath('_exts'))
# -- General configuration ------------------------------------------------
# If your documentation needs a minimal Sphinx version, state it here.
needs_sphinx = '1.3'
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = [
'sphinx.ext.autodoc',
'sphinx.ext.intersphinx',
'sphinx.ext.napoleon',
'pyspecific'
]
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
# The suffix(es) of source filenames.
# You can specify multiple suffix as a list of string:
# source_suffix = ['.rst', '.md']
source_suffix = '.rst'
# The encoding of source files.
#source_encoding = 'utf-8-sig'
# The master toctree document.
master_doc = 'index'
# General information about the project.
project = 'AIO Disque'
copyright = '2016, Xavier Barbosa'
author = 'Xavier Barbosa'
# The version info for the project you're documenting, acts as replacement for
# |version| and |release|, also used in various other places throughout the
# built documents.
#
# The short X.Y version.
version = '1.0'
# The full version, including alpha/beta/rc tags.
release = '1.0'
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
#
# This is also used if you do content translation via gettext catalogs.
# Usually you set "language" from the command line for these cases.
language = None
autodoc_member_order = 'bysource'
# There are two options for replacing |today|: either, you set today to some
# non-false value, then it is used:
#today = ''
# Else, today_fmt is used as the format for a strftime call.
#today_fmt = '%B %d, %Y'
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
exclude_patterns = ['_build']
# The reST default role (used for this markup: `text`) to use for all
# documents.
#default_role = None
# If true, '()' will be appended to :func: etc. cross-reference text.
#add_function_parentheses = True
# If true, the current module name will be prepended to all description
# unit titles (such as .. function::).
#add_module_names = True
# If true, sectionauthor and moduleauthor directives will be shown in the
# output. They are ignored by default.
#show_authors = False
# The name of the Pygments (syntax highlighting) style to use.
pygments_style = 'sphinx'
# A list of ignored prefixes for module index sorting.
#modindex_common_prefix = []
# If true, keep warnings as "system message" paragraphs in the built documents.
#keep_warnings = False
# If true, `todo` and `todoList` produce output, else they produce nothing.
todo_include_todos = False
# -- Options for HTML output ----------------------------------------------
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
html_theme = 'alabaster'
# Theme options are theme-specific and customize the look and feel of a theme
# further. For a list of options available for each theme, see the
# documentation.
#html_theme_options = {}
# Add any paths that contain custom themes here, relative to this directory.
#html_theme_path = []
# The name for this set of Sphinx documents. If None, it defaults to
# "<project> v<release> documentation".
#html_title = None
# A shorter title for the navigation bar. Default is the same as html_title.
#html_short_title = None
# The name of an image file (relative to this directory) to place at the top
# of the sidebar.
#html_logo = None
# The name of an image file (within the static path) to use as favicon of the
# docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32
# pixels large.
#html_favicon = None
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ['_static']
# Add any extra paths that contain custom files (such as robots.txt or
# .htaccess) here, relative to this directory. These files are copied
# directly to the root of the documentation.
#html_extra_path = []
# If not '', a 'Last updated on:' timestamp is inserted at every page bottom,
# using the given strftime format.
#html_last_updated_fmt = '%b %d, %Y'
# If true, SmartyPants will be used to convert quotes and dashes to
# typographically correct entities.
#html_use_smartypants = True
# Custom sidebar templates, maps document names to template names.
#html_sidebars = {}
# Additional templates that should be rendered to pages, maps page names to
# template names.
#html_additional_pages = {}
# If false, no module index is generated.
#html_domain_indices = True
# If false, no index is generated.
#html_use_index = True
# If true, the index is split into individual pages for each letter.
#html_split_index = False
# If true, links to the reST sources are added to the pages.
#html_show_sourcelink = True
# If true, "Created using Sphinx" is shown in the HTML footer. Default is True.
#html_show_sphinx = True
# If true, "(C) Copyright ..." is shown in the HTML footer. Default is True.
#html_show_copyright = True
# If true, an OpenSearch description file will be output, and all pages will
# contain a <link> tag referring to it. The value of this option must be the
# base URL from which the finished HTML is served.
#html_use_opensearch = ''
# This is the file name suffix for HTML files (e.g. ".xhtml").
#html_file_suffix = None
# Language to be used for generating the HTML full-text search index.
# Sphinx supports the following languages: