O i9dZddlZddlZddlZddlmZmZddlmZddl m Z ddl m Z ddl mZdZd ZGd d ZGd d eZGddeZdS)u:class:`.RateLimiter` and :class:`.AsyncRateLimiter` allow to perform bulk operations while gracefully handling error responses and adding delays when needed. In the example below a delay of 1 second (``min_delay_seconds=1``) will be added between each pair of ``geolocator.geocode`` calls; all :class:`geopy.exc.GeocoderServiceError` exceptions will be retried (up to ``max_retries`` times):: import pandas as pd df = pd.DataFrame({'name': ['paris', 'berlin', 'london']}) from geopy.geocoders import Nominatim geolocator = Nominatim(user_agent="specify_your_app_name_here") from geopy.extra.rate_limiter import RateLimiter geocode = RateLimiter(geolocator.geocode, min_delay_seconds=1) df['location'] = df['name'].apply(geocode) df['point'] = df['location'].apply(lambda loc: tuple(loc.point) if loc else None) This would produce the following DataFrame:: >>> df name location \ 0 paris (Paris, Île-de-France, France métropolitaine, ... 1 berlin (Berlin, 10117, Deutschland, (52.5170365, 13.3... 2 london (London, Greater London, England, SW1A 2DU, UK... point 0 (48.8566101, 2.3514992, 0.0) 1 (52.5170365, 13.3888599, 0.0) 2 (51.5073219, -0.1276474, 0.0) To pass extra options to the `geocode` call:: from functools import partial df['location'] = df['name'].apply(partial(geocode, language='de')) To see a progress bar:: from tqdm import tqdm tqdm.pandas() df['location'] = df['name'].progress_apply(geocode) Before using rate limiting classes, please consult with the Geocoding service ToS, which might explicitly consider bulk requests (even throttled) a violation. N)chaincount)sleep default_timer)GeocoderServiceError)logger)AsyncRateLimiter RateLimitercRtdt|DdgS)z-list(_is_last_gen(2)) -> [False, False, True]c3K|]}dVdS)FN).0_s P/srv/django_bis/venv311/lib/python3.11/site-packages/geopy/extra/rate_limiter.py z_is_last_gen..Bs"..A%......T)rrange)rs r _is_last_genr@s) ..u... 7 77rc6eZdZdZefZdZdZdZdZ dZ dS)BaseRateLimiterz9Base Rate Limiter class for both sync and async versions.c||_||_||_||_|dksJt j|_d|_dS)Nr)min_delay_seconds max_retriesswallow_exceptionsreturn_value_on_exception threadingLock_lock _last_call)selfrrrrs r__init__zBaseRateLimiter.__init__JsQ"3&"4)B&a^%% rctSNr)r!s r_clockzBaseRateLimiter._clock\s rc#K |j5|}|j||_ ddddS||jz }|j|z }|dkr||_ ddddS dddn #1swxYwY|V)NTr)rr%r r)r!clockseconds_since_last_callwaits r_acquire_request_slot_genz)BaseRateLimiter._acquire_request_slot_gen_s*    ?*&+DO         +0$/*A'-0GG199&+DO                       JJJ s$A8"A88A<?A<c #"Kttt|jD]]\}} |VdS#|j$rD|rdVn:t jt|jdz||j||ddVYXYZwxYwdS)NTzB caught an error, retrying (%s/%s tries). Called with (*%r, **%r).exc_infoF) ziprrr_retry_exceptionsr warningtype__name__)r!argskwargsi is_last_trys r _retries_genzBaseRateLimiter._retries_gens!%''<8H+I+IJJ  NA{ $#)    JJJJNT +/BB(!% KKKHJ   s>A B  B c|jr:tjt|jdz|j||d|jS)Nz> swallowed an error after %r retries. Called with (*%r, **%r).Tr,)rr r0r1r2rr)r!r3r4s r _handle_exczBaseRateLimiter._handle_excsY  "  NT #'++      1 1 rN) r2 __module__ __qualname____doc__rr/r"r%r*r7r9rrrrrEsoCC-/$$$$L.     rrcBeZdZdZddddddfd Zd Zd Zd ZxZS) r aThis is a Rate Limiter implementation for synchronous functions (like geocoders with the default :class:`geopy.adapters.BaseSyncAdapter`). Examples:: from geopy.extra.rate_limiter import RateLimiter from geopy.geocoders import Nominatim geolocator = Nominatim(user_agent="specify_your_app_name_here") search = ["moscow", "paris", "berlin", "tokyo", "beijing"] geocode = RateLimiter(geolocator.geocode, min_delay_seconds=1) locations = [geocode(s) for s in search] search = [ (55.47, 37.32), (48.85, 2.35), (52.51, 13.38), (34.69, 139.40), (39.90, 116.39) ] reverse = RateLimiter(geolocator.reverse, min_delay_seconds=1) locations = [reverse(s) for s in search] RateLimiter class is thread-safe. If geocoding service's responses are slower than `min_delay_seconds`, then you can benefit from parallelizing the work:: import concurrent.futures geolocator = OpenMapQuest(api_key="...") geocode = RateLimiter(geolocator.geocode, min_delay_seconds=1/20) with concurrent.futures.ThreadPoolExecutor() as e: locations = list(e.map(geocode, search)) .. versionchanged:: 2.0 Added thread-safety support. @TNrrerror_wait_secondsrrct||||||_||_||ksJ|dksJdSa  :param callable func: A function which should be wrapped by the rate limiter. :param float min_delay_seconds: Minimum delay in seconds between the wrapped ``func`` calls. To convert :abbr:`RPS (Requests Per Second)` rate to ``min_delay_seconds`` you need to divide 1 by RPS. For example, if you need to keep the rate at 20 RPS, you can use ``min_delay_seconds=1/20``. :param int max_retries: Number of retries on exceptions. Only :class:`geopy.exc.GeocoderServiceError` exceptions are retried -- others are always re-raised. ``max_retries + 1`` requests would be performed at max per query. Set ``max_retries=0`` to disable retries. :param float error_wait_seconds: Time to wait between retries after errors. Must be greater or equal to ``min_delay_seconds``. :param bool swallow_exceptions: Should an exception be swallowed after retries? If not, it will be re-raised. If yes, the ``return_value_on_exception`` will be returned. :param return_value_on_exception: Value to return on failure when ``swallow_exceptions=True``. )rrrrrNsuperr"funcrBr!rGrrrBrr __class__s rr"zRateLimiter.__init__mR /#1&?     "4!%66666arcxtjt|jdz|t |dSNz sleep(%r))r debugr1r2rr!secondss r_sleepzRateLimiter._sleeps1 T$ZZ(<7AAA grc^|D]}||dSr$r*rPr!r)s r_acquire_request_slotz!RateLimiter._acquire_request_slot s<2244  D KK      rc|||}|D]}| |j|i|}tj|rt d|cS#|j$rV}||r|||cYd}~cS| |j Yd}~d}~wwxYwtd)NzoAn async awaitable has been passed to `RateLimiter`. Use `AsyncRateLimiter` instead, which supports awaitables.Should not have been reached) r7rTrGinspect isawaitable ValueErrorr/throwr9rPrB RuntimeError)r!r3r4genrreses r__call__zRateLimiter.__call__ sf-- 5 5A  & & ( ( ( 5di000&s++$U ) 5 5 599Q<<:++D&9999999999 D344444444  5 9:::s#1A$$ C.*B?C B??C r2r:r;r<r"rPrTr_ __classcell__rIs@rr r s##R"&2 2 2 2 2 2 2 h;;;;;;;rr cBeZdZdZddddddfd Zd Zd Zd ZxZS) r aThis is a Rate Limiter implementation for asynchronous functions (like geocoders with :class:`geopy.adapters.BaseAsyncAdapter`). Examples:: from geopy.adapters import AioHTTPAdapter from geopy.extra.rate_limiter import AsyncRateLimiter from geopy.geocoders import Nominatim async with Nominatim( user_agent="specify_your_app_name_here", adapter_factory=AioHTTPAdapter, ) as geolocator: search = ["moscow", "paris", "berlin", "tokyo", "beijing"] geocode = AsyncRateLimiter(geolocator.geocode, min_delay_seconds=1) locations = [await geocode(s) for s in search] search = [ (55.47, 37.32), (48.85, 2.35), (52.51, 13.38), (34.69, 139.40), (39.90, 116.39) ] reverse = AsyncRateLimiter(geolocator.reverse, min_delay_seconds=1) locations = [await reverse(s) for s in search] AsyncRateLimiter class is safe to use across multiple concurrent tasks. If geocoding service's responses are slower than `min_delay_seconds`, then you can benefit from parallelizing the work:: import asyncio async with OpenMapQuest( api_key="...", adapter_factory=AioHTTPAdapter ) as geolocator: geocode = AsyncRateLimiter(geolocator.geocode, min_delay_seconds=1/20) locations = await asyncio.gather(*(geocode(s) for s in search)) .. versionadded:: 2.0 r>r?r@TNrAct||||||_||_||ksJ|dksJdSrDrErHs rr"zAsyncRateLimiter.__init__LrJrcKtjt|jdz|t j|d{VdSrL)r rMr1r2asynciorrNs rrPzAsyncRateLimiter._sleepsN T$ZZ(<7AAAmG$$$$$$$$$$$rcnK|D]}||d{VdSr$rRrSs rrTz&AsyncRateLimiter._acquire_request_slotsR2244 $ $D++d## # # # # # # # # $ $rcK|||}|D]}|d{V |j|i|d{VcS#|j$r\}||r|||cYd}~cS||jd{VYd}~d}~wwxYwtd)NrV) r7rTrGr/rZr9rPrBr[)r!r3r4r\rr^s rr_zAsyncRateLimiter.__call__s(f-- ; ;A,,.. . . . . . . . ;&TY777777777777) ; ; ;99Q<<:++D&9999999999kk$"9::::::::::::::  ; 9:::s#A  B3*B.B3 B..B3r`rbs@rr r "s''Z"&2 2 2 2 2 2 2 h%%%$$$ ; ; ; ; ; ; ;rr )r<rfrWr itertoolsrrtimertimeitr geopy.excr geopy.utilr __all__rrr r rrrrosF00d"""""""" ****** -888 ccccccccLt;t;t;t;t;/t;t;t;nr;r;r;r;r;r;r;r;r;r;r