GitLab

I only learned this from https://github.com/celery/celery/issues/1958 which requests a doc update to make this clearer.

test on win32 py3.9 with pycurl windows wheels from https://www.lfd.uci.edu/~gohlke/pythonlibs/ (#6875)

* use windows wheels from https://www.lfd.uci.edu/~gohlke/pythonlibs/

you're not supposed to use the wheels directly so I made my own mirror
on github pages
If you merge this I'll need you to move the repo into the celery org

* use find-links

* pycurl direct reference

* fix platform_system typo

* unexeclude win32 pypy and 3.10

* Update tox.ini

* Revert "unexeclude win32 pypy and 3.10"

This reverts commit 6bb7e8a980f3839f310607c767c8a97f563ca345.

* try simple repo

* use the celery.github.io wheelhouse

updates:
- [github.com/asottile/pyupgrade: v2.23.0 → v2.23.1](https://github.com/asottile/pyupgrade/compare/v2.23.0...v2.23.1)
- [github.com/pycqa/isort: 5.9.2 → 5.9.3](https://github.com/pycqa/isort/compare/5.9.2...5.9.3)

-Ofair is now the default scheduling strategy as of v4.0: https://github.com/celery/celery/blob/8ebcce1523d79039f23da748f00bec465951de2a/docs/history/whatsnew-4.0.rst#ofair-is-now-the-default-scheduling-strategy

updates:
- [github.com/asottile/pyupgrade: v2.21.2 → v2.23.0](https://github.com/asottile/pyupgrade/compare/v2.21.2...v2.23.0)
- https://gitlab.com/pycqa/flake8 → https://github.com/PyCQA/flake8Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* unignore f821

* defer celery imports in celery pytest plugin

* create github action for windows

* increase tox verbosity

* configure pip caching/requirements

* Update .github/workflows/windows.yml

* define kombu sqs passthrough dep

* drop 3.9 from windows due to pycurl

* skip test_check_privileges_suspicious_platform[accept_content0] on win32, py38+

* fails on py38+ win32

* bump the maxfail a bit to get more error context

* xfail all py3.8+ windows tests

* re-enable -v

* pytest.raises does not raise AssertionError

https://github.com/pytest-dev/pytest/issues/8928

* more xfails

* merge windows workflow into python-package

* only install apt packages on ubuntu-*

* bust pip cache with matrix.os

* step.if doesn't need {{

* Update python-package.yml

* Windows is never considerred a sus platform

this is because Microsft is beyond reproach

* fix merge resolution error

* doc: Amend IRC network link to Libera

Ref #6811

* Update README.rst
Co-authored-by: Thomas Grainger <tagrain@gmail.com>
Co-authored-by: Asif Saif Uddin <auvipy@gmail.com>
Co-authored-by: Thomas Grainger <tagrain@gmail.com>

* relaxed click version

* fix get_default

* pre-check WorkersPool click.Choice type before calling super

https://github.com/pallets/click/issues/1898#issuecomment-841546735

* apply pre-commit run --all-files
Co-authored-by: Asif Saif Uddin <auvipy@gmail.com>

* configure pre-commit (from twisted)

* remove black

* run pre-commit in ci

* configure isort with pre-commit

* configure pre-commit in tox

* allow E203 for black support in the future

* update contributing guide

* apply pyupgrade

* apply isort

* apply yes-qa

Fixes #6855.

A connection loss flushes the asynpool (See https://github.com/celery/celery/blob/117cd9ca410e8879f71bd84be27b8e69e462c56a/celery/worker/consumer/consumer.py#L414).
This is expected as these jobs cannot be completed anymore.
However, jobs which have not been accepted yet (that is, they are not running yet) are cancelled. This only works if the synack keyword argument is set to True.
In our case, it isn't and therefore the jobs remain in the pool's cache forever.

This is a memory leak which we have now resolved by discarding the job (which clears it from the cache) as they will never be cancelled.

Update default `worker_task_log_format` value

Allow to set "worker_concurrency" option through
"user_preload_options" signal mechanism.

Current behaviour:

1. "click.option" decorator for "--concurrency" option is executed,
   its callback returns "0" when evaluating "value or
   ctx.obj.app.conf.worker_concurrency" (None or 0). This default "0"
   comes from "app.defaults".

2. Celery "user_preload_options" signal is processed, then
   "app.conf.worker_concurrency" value is correctly updated through
   "Settings.update".

3. Celery "worker.worker.WorkController.setup_defaults" kicks off
   and "concurrency" attribute is resolved with
   "either('worker_concurrency', concurrency)"

4. "either" method (app.base) chains calls to "first" function with
   "None" as predicate (returns the first item that's not "None"),
   in our case "first(None, defaults)" (defaults=(0,)) will take
   precedence and and "0" will be returned, whatever value is in
   "app.conf.worker_concurrency".

This fix changes "worker_concurrency" default from "0" to "None"
allowing "either" method to correctly resolve in favor
of "app.conf.worker_concurrency" value.

The final value used as concurrency is resolved in "worker.worker"
with conditional "if not self.concurrency" thus having "None"
as default value for "self.concurrency" doesn't break things.

Fixes #6836

* Fix Open Collective donate button

Fixes #6828

* Use OpenCollective anchor button

- Replace OpenCollective button script with an <a> tag. The button
script imposes a fixed width of 300px which makes it too big and
out of place relative to neighbouring HTML elements.
Co-authored-by: Lewis Kabui <lewisemm@users.noreply.github.com>

* Add test of backend usage by threads

Add simple test with embedded worker that checks
backend instance usage by threads. According
merge request #6416 backends should be thread local.

* Fix backend captures in the `celery.app.trace.build_tracer`

Fix backend capturing by closure during task creation in the function `celery.app.trace.build_tracer`, as different threads may create and use celery task. It complement changes in the pull request #6416.

* Fix flake8 errors

Fix flake8 errors from Celery/lint github workflow step

* Fix #6844 by allowing safe (i.e. skip arg derserialization) queries via app.inspect().active().

* Fix default active arg test expectation.

* Fix test asserting broken behaviour (arg/kwarg deserialization occuring when safe=True).
Co-authored-by: Damir Jungic <djungic@cisco.com>

* Get body_type independently to handle cases where body.type does not exist due to tasks being created via Signatures

* body.get() was returning None always, must getattr() and catch the NotRegistered Error if the app that generated the task is not the app that owns the task

* flake8 fix for too many blank lines

* Create changerelease.yml

* Update changerelease.yml

* Update changerelease.yml

* Update changerelease.yml

* Update changerelease.yml

* Update changerelease.yml

* Update changerelease.yml

* Update changerelease.yml

* Update changerelease.yml

* Update changerelease.yml

* Update changerelease.yml

* Update changerelease.yml

* Add workflow permissions

* Add Python 3.10 support.
* Use the dev release for now.
* Include deps for 3.10.
* Bump moto to support Python 3.10.
* Currently, eventlet is not supported by 3.10.
* Skip if eventlet not found.

* Test 3.10 using tox.
* Try tox-gh-actions.
* Map python versions to tox environments.
* Allow the 3.10 job to fail for now.

 * reintroduce sys.argv default behaviour for "start" (as was
   commented for "worker_main" in https://github.com/celery/celery/pull/6481#discussion_r524048986
 * reintroduce docstrings for "start" and "worker_main" methods
 * reintroduce and adapt tests for "start" and "worker_main"

Programmatic start (code and unittests) was removed due to
01651d2f and reintroduced
in #6481.

Resolves: #6730
Relates: #6481 #6404

Mock's aren't mapping-like enough for this to work in pypy3, but
MagicMocks are.