U +if @sdZddlmZddlmZmZmZmZddlZddl Z ddl m Z m Z mZddlmZddlmZddlmZdd lmZdd lmZmZmZddlmmZdd lmZm Z ddl!mm"Z#dd l$m%Z%dd l&m'Z'ddl(m)Z)m*Z*ddl+m,Z,m-Z-m.Z.ddl/m0Z0ddl1m2Z2er*ddl3m4Z4GdddZ5eGdddZ6d)dddddddddddZ7dd d!d"Z8d#d$d%d&Z9d'd(Z:dS)*z] Provide user facing operators for doing the split part of the split-apply-combine paradigm. ) annotations) TYPE_CHECKINGAnyHashablefinalN) ArrayLikeNDFrameTnpt)InvalidIndexError)cache_readonly)find_stack_level)sanitize_to_nanoseconds)is_categorical_dtype is_list_like is_scalar) CategoricalExtensionArray) DataFrame)ops)recode_for_groupbyrecode_from_groupby)CategoricalIndexIndex MultiIndex)Series) pprint_thing)NDFramecseZdZUdZded<ded<ded<ded<ded <d Zd ed <fd dZd(ddddddZee ddddZ d)ddddddZ ed*ddd d!d"Z ee d#d$Z ed%dd&d'ZZS)+Grouperan A Grouper allows the user to specify a groupby instruction for an object. This specification will select a column via the key parameter, or if the level and/or axis parameters are given, a level of the index of the target object. If `axis` and/or `level` are passed as keywords to both `Grouper` and `groupby`, the values passed to `Grouper` take precedence. Parameters ---------- key : str, defaults to None Groupby key, which selects the grouping column of the target. level : name/number, defaults to None The level for the target index. freq : str / frequency object, defaults to None This will groupby the specified frequency if the target selection (via key or level) is a datetime-like object. For full specification of available frequencies, please see `here `_. axis : str, int, defaults to 0 Number/name of the axis. sort : bool, default to False Whether to sort the resulting labels. closed : {'left' or 'right'} Closed end of interval. Only when `freq` parameter is passed. label : {'left' or 'right'} Interval boundary to use for labeling. Only when `freq` parameter is passed. convention : {'start', 'end', 'e', 's'} If grouper is PeriodIndex and `freq` parameter is passed. base : int, default 0 Only when `freq` parameter is passed. For frequencies that evenly subdivide 1 day, the "origin" of the aggregated intervals. For example, for '5min' frequency, base could range from 0 through 4. Defaults to 0. .. deprecated:: 1.1.0 The new arguments that you should use are 'offset' or 'origin'. loffset : str, DateOffset, timedelta object Only when `freq` parameter is passed. .. deprecated:: 1.1.0 loffset is only working for ``.resample(...)`` and not for Grouper (:issue:`28302`). However, loffset is also deprecated for ``.resample(...)`` See: :class:`DataFrame.resample` origin : Timestamp or str, default 'start_day' The timestamp on which to adjust the grouping. The timezone of origin must match the timezone of the index. If string, must be one of the following: - 'epoch': `origin` is 1970-01-01 - 'start': `origin` is the first value of the timeseries - 'start_day': `origin` is the first day at midnight of the timeseries .. versionadded:: 1.1.0 - 'end': `origin` is the last value of the timeseries - 'end_day': `origin` is the ceiling midnight of the last day .. versionadded:: 1.3.0 offset : Timedelta or str, default is None An offset timedelta added to the origin. .. versionadded:: 1.1.0 dropna : bool, default True If True, and if group keys contain NA values, NA values together with row/column will be dropped. If False, NA values will also be treated as the key in groups. .. versionadded:: 1.2.0 Returns ------- A specification for a groupby instruction Examples -------- Syntactic sugar for ``df.groupby('A')`` >>> df = pd.DataFrame( ... { ... "Animal": ["Falcon", "Parrot", "Falcon", "Falcon", "Parrot"], ... "Speed": [100, 5, 200, 300, 15], ... } ... ) >>> df Animal Speed 0 Falcon 100 1 Parrot 5 2 Falcon 200 3 Falcon 300 4 Parrot 15 >>> df.groupby(pd.Grouper(key="Animal")).mean() Speed Animal Falcon 200.0 Parrot 10.0 Specify a resample operation on the column 'Publish date' >>> df = pd.DataFrame( ... { ... "Publish date": [ ... pd.Timestamp("2000-01-02"), ... pd.Timestamp("2000-01-02"), ... pd.Timestamp("2000-01-09"), ... pd.Timestamp("2000-01-16") ... ], ... "ID": [0, 1, 2, 3], ... "Price": [10, 20, 30, 40] ... } ... ) >>> df Publish date ID Price 0 2000-01-02 0 10 1 2000-01-02 1 20 2 2000-01-09 2 30 3 2000-01-16 3 40 >>> df.groupby(pd.Grouper(key="Publish date", freq="1W")).mean() ID Price Publish date 2000-01-02 0.5 15.0 2000-01-09 2.0 30.0 2000-01-16 3.0 40.0 If you want to adjust the start of the bins based on a fixed timestamp: >>> start, end = '2000-10-01 23:30:00', '2000-10-02 00:30:00' >>> rng = pd.date_range(start, end, freq='7min') >>> ts = pd.Series(np.arange(len(rng)) * 3, index=rng) >>> ts 2000-10-01 23:30:00 0 2000-10-01 23:37:00 3 2000-10-01 23:44:00 6 2000-10-01 23:51:00 9 2000-10-01 23:58:00 12 2000-10-02 00:05:00 15 2000-10-02 00:12:00 18 2000-10-02 00:19:00 21 2000-10-02 00:26:00 24 Freq: 7T, dtype: int64 >>> ts.groupby(pd.Grouper(freq='17min')).sum() 2000-10-01 23:14:00 0 2000-10-01 23:31:00 9 2000-10-01 23:48:00 21 2000-10-02 00:05:00 54 2000-10-02 00:22:00 24 Freq: 17T, dtype: int64 >>> ts.groupby(pd.Grouper(freq='17min', origin='epoch')).sum() 2000-10-01 23:18:00 0 2000-10-01 23:35:00 18 2000-10-01 23:52:00 27 2000-10-02 00:09:00 39 2000-10-02 00:26:00 24 Freq: 17T, dtype: int64 >>> ts.groupby(pd.Grouper(freq='17min', origin='2000-01-01')).sum() 2000-10-01 23:24:00 3 2000-10-01 23:41:00 15 2000-10-01 23:58:00 45 2000-10-02 00:15:00 45 Freq: 17T, dtype: int64 If you want to adjust the start of the bins with an `offset` Timedelta, the two following lines are equivalent: >>> ts.groupby(pd.Grouper(freq='17min', origin='start')).sum() 2000-10-01 23:30:00 9 2000-10-01 23:47:00 21 2000-10-02 00:04:00 54 2000-10-02 00:21:00 24 Freq: 17T, dtype: int64 >>> ts.groupby(pd.Grouper(freq='17min', offset='23h30min')).sum() 2000-10-01 23:30:00 9 2000-10-01 23:47:00 21 2000-10-02 00:04:00 54 2000-10-02 00:21:00 24 Freq: 17T, dtype: int64 To replace the use of the deprecated `base` argument, you can now use `offset`, in this example it is equivalent to have `base=2`: >>> ts.groupby(pd.Grouper(freq='17min', offset='2min')).sum() 2000-10-01 23:16:00 0 2000-10-01 23:33:00 9 2000-10-01 23:50:00 36 2000-10-02 00:07:00 39 2000-10-02 00:24:00 24 Freq: 17T, dtype: int64 intaxisboolsortdropna Index | None _gpr_index_grouper)keylevelfreqrr!ztuple[str, ...] _attributescs6|ddk r*ddlm}t||d|}t|S)Nr(r) TimeGrouper)origin)getZpandas.core.resampler*!_check_deprecated_resample_kwargssuper__new__)clsargskwargsr* __class__N/home/mars/bis/venv/lib/python3.8/site-packages/pandas/core/groupby/grouper.pyr/ s   zGrouper.__new__NrFT)rr!r"cCsR||_||_||_||_||_d|_d|_d|_d|_d|_ d|_ d|_ ||_ dSN) r&r'r(rr!grouperr$objindexerbinnerr%_indexerr")selfr&r'r(rr!r"r5r5r6__init__s zGrouper.__init__rreturncCs|j}|dkrtd|S)Nz1_set_grouper must be called before ax is accessed)r$ ValueError)r=indexr5r5r6ax,sz Grouper.axrz%tuple[Any, ops.BaseGrouper, NDFrameT])r9validater@c CsH||t|j|jg|j|j|j||jd\|_}|_|j |j|jfS)z Parameters ---------- obj : Series or DataFrame validate : bool, default True if True, validate the grouper Returns ------- a tuple of binner, grouper, obj (possibly sorted) )rr'r!rDr") _set_grouper get_grouperr9r&rr'r!r"r8r;)r=r9rD_r5r5r6 _get_grouper4s  zGrouper._get_grouperr)r9r!c Cs|dk s t|jdk r(|jdk r(td|jdkrB|j|_|j|_|jdk r|j}t|jdd|krt |t r|jdk s|t|jdk r|j }|j |}| |j }q|j |j }n*||jkrtd|dt|||d}nn||j}|jdk rT|j}t |tr4||}t|||j|d}n |d|jfkrTtd|d |jsb|r|js|jj d d d }|_| |}|j ||jd }||_||_|jS)a% given an object and the specifications, setup the internal grouper for this particular specification Parameters ---------- obj : Series or DataFrame sort : bool, default False whether the resulting grouper should be sorted Nz2The Grouper cannot specify both a key and a level!namezThe grouper name z is not foundrIrz The level z is not validZ mergesortfirst)kindZ na_positionr)AssertionErrorr&r'rAr%r$r:r<getattr isinstancerZargsortZtakerBZ _info_axisKeyErrorr _get_axisrrZ_get_level_numberZ_get_level_valuesnamesrIr!Z is_monotonicarrayr9) r=r9r!r&Zreverse_indexerZ unsorted_axrCr'r:r5r5r6rEUsN             zGrouper._set_groupercCs|jjSr7)r8groupsr=r5r5r6rUszGrouper.groupsstrcs8fddjD}d|}tj}|d|dS)Nc3s4|],}t|dk r|dtt|VqdS)N=)rOrepr).0 attr_namerVr5r6 sz#Grouper.__repr__..z, ())r)jointype__name__)r=Z attrs_listattrscls_namer5rVr6__repr__s    zGrouper.__repr__)NNNrFT)T)F)ra __module__ __qualname____doc____annotations__r)r/r>rpropertyrCrHrErUrd __classcell__r5r5r3r6r:s8 J  !L rc@s&eZdZUdZdZded<dZded<ded<d ed <d ed <d5d ddddddddZddddZddZ e ddddZ e ddddZ e ddd d!Ze d"dd#d$Ze d%dd&d'Ze d(dd)d*Ze d dd+d,Ze d dd-d.Ze d/dd0d1Ze d2dd3d4ZdS)6GroupingaJ Holds the grouping information for a single key Parameters ---------- index : Index grouper : obj : DataFrame or Series name : Label level : observed : bool, default False If we are a Categorical, use the observed values in_axis : if the Grouping is a column in self.obj and hence among Groupby.exclusions list Returns ------- **Attributes**: * indices : dict of {group -> index_list} * codes : ndarray, group codes * group_index : unique groups * groups : dict of {group -> label_list} Nznp.ndarray | None_codesr# _group_indexr _passed_categoricalzCategorical | None _all_grouperr_indexTFzNDFrame | None)rBr9r!observedin_axisr"c Cs||_||_t|||_d|_||_||_||_||_||_ ||_ d|_ |j } | dk rz|j} |j | | d\|_|_|_n6t|jtr|jdk st|jj|jdd\} } } | |_| }t| tjr||_nt|| jjd|_nt|jr d|_ t|j||\|_|_nt|jttttjfst|jdddkr^|jpLt t!|j}t"d|d |#|j|_t$|jd rt%|jt%|kst&|j}d |}d|_t|t|jtjrt'|j|_dS) NF)r'rDrJTndim Grouper for '' not 1-dimensional__len__z9Grouper result violates len(labels) == len(data) result: )(r' _orig_grouper_convert_groupergrouping_vectorrorp_sortr9 _observedrr_dropnarn_ilevelZ_get_grouper_for_levelrlrmrPrrNrHrZ BinGrouperr result_indexrIrrrrnpndarrayrOrWr`rAmaphasattrlenrr )r=rBr8r9r'r!rqrrr"ilevelZmapperrGZ newgrouperZnewobjngtZgrpererrmsgr5r5r6r>sr          zGrouping.__init__rWr?cCsd|jdS)Nz Grouping(r^rJrVr5r5r6rd4szGrouping.__repr__cCs t|jSr7)iterindicesrVr5r5r6__iter__7szGrouping.__iter__rcCsb|j}|dk r|jj|St|jttfr2|jjSt|jt j rJ|jj jSt|jtr^|jjSdSr7) rrprSrPryrrrIr{r BaseGrouperr)r=rr5r5r6rI:s   z Grouping.namez int | NonecCsL|j}|dkrdSt|tsH|j}||jkrrdrr rIrrirrrrrrrrUr5r5r5r6rksH   ` #rkTFrrr z5tuple[ops.BaseGrouper, frozenset[Hashable], NDFrameT])r9rr!rqmutatedrDr"r@c  s4|} |dk rt| trXt|r8t|dkr8|d}|dkrt|r| |}d}nt|rt|} | dkrz|d}n| dkrtdntdt|trʈ|j |krtd|d |n|dks|dkrtd d}| }t|t r6|j d d \} } |j dkr"| tfS| t|j hfSnt|tjrP|tfSt|tsh|g} d }n|} t| t| k}td d | D}tdd | D}tdd | D}|s0|s0|s0|r0|dkr0ttrtfdd | D}n&ttsttfdd | D}|s0t| g} t|ttfr^|dkrXdgt|} |}n|gt| }g}t}ddfdd }ddfdd }t| |D]\}}||rd}||j n||rZ|kr.|rj||dd||}}}|jdkr"td|d||n*j||drPd |d}}}nt |n.t|t r|j dk r||j d }nd }t|t!st!| ||||||dn|}|"|qt|dkrtrtdn2t|dkr|"t!t#gddt$j%gt$j&dtj| ||||d} | t|fS) a Create and return a BaseGrouper, which is an internal mapping of how to create the grouper indexers. This may be composed of multiple Grouping objects, indicating multiple groupers Groupers are ultimately index mappings. They can originate as: index mappings, keys to columns, functions, or Groupers Groupers enable local references to axis,level,sort, while the passed in axis, level, and sort are 'global'. This routine tries to figure out what the passing in references are and then creates a Grouping for each one, combined into a BaseGrouper. If observed & we have a categorical grouper, only show the observed values. If validate, then check for key/level overlaps. NrurzNo group keys passed!z*multiple levels only valid with MultiIndexz level name z is not the name of the rz2level > 0 or level < -1 only valid with MultiIndexFrscss |]}t|pt|tVqdSr7)callablerPdictrZgr5r5r6r\"szget_grouper..css|]}t|ttfVqdSr7)rPrrkrr5r5r6r\#scss$|]}t|tttttjfVqdSr7)rPlisttuplerrrrrr5r5r6r\$sc3s$|]}|jkp|jjkVqdSr7)columnsrBrSrr9r5r6r\1sc3s|]}|jjkVqdSr7)rBrSrrr5r6r\6sr r?c sBt|s>jd}z||Wntttfk r<YdSXdS)NrFT)_is_label_likeZaxesZget_locrQ TypeErrorr )r&itemsrr5r6 is_in_axisFs zget_grouper..is_in_axisc s@t|dsdSz||jkWStttfk r:YdSXdS)NrIF)rrIrQ IndexErrorr )gprrr5r6 is_in_objSs  zget_grouper..is_in_objTrMrvrw)r9r'r!rqrrr"r)Zdtype)r!rr")'rRrPrrrrZget_level_valuesrArWrIZ_get_axis_namerrHr& frozensetrrranyrallrrNcomasarray_tuplesafersetzipaddZ_check_label_or_level_ambiguityrtZ_is_level_referencerQrkappendrrrTZintp)r9r&rr'r!rqrrDr"Z group_axisZnlevelsr;r8keysZmatch_axis_lengthZ any_callableZ any_groupersZ any_arraylikeZall_in_columns_indexZlevelsZ groupingsZ exclusionsrrrrrrIZpingr5rr6rFs!                           $rFr?cCst|ttfp|dk ot|Sr7)rPrWrr)valr5r5r6rsrrrMcCst|tr|jSt|tr:|j|r,|jS||jSn^t|trJ|jSt|t t t t t jfrt|t|krxtdt|t t frt|}|S|SdS)Nz$Grouper and axis must be same length)rPrr,rrBequalsrZreindexrrrrrrrrrArr)rr8r5r5r6rzs     rzcCsH|dddk r"tjdttd|dddk rDtjdttddS)a Check for use of deprecated parameters in ``resample`` and related functions. Raises the appropriate warnings if these parameters are detected. Only sets an approximate ``stacklevel`` for the warnings (see #37603, #36629). Parameters ---------- kwargs : dict Dictionary of keyword arguments to check for deprecated parameters. origin : object From where this function is being called; either Grouper or TimeGrouper. Used to determine an approximate stacklevel. baseNz'base' in .resample() and in Grouper() is deprecated. The new arguments that you should use are 'offset' or 'origin'. >>> df.resample(freq="3s", base=2) becomes: >>> df.resample(freq="3s", offset="2s") ) stacklevelZloffseta'loffset' in .resample() and in Grouper() is deprecated. >>> df.resample(freq="3s", loffset="8H") becomes: >>> from pandas.tseries.frequencies import to_offset >>> df = df.resample(freq="3s").mean() >>> df.index = df.index.to_timestamp() + to_offset("8H") )r,warningswarn FutureWarningr )r2r+r5r5r6r-s r-)NrNTFFTT);rg __future__rtypingrrrrrnumpyrZpandas._typingrrr Z pandas.errorsr Zpandas.util._decoratorsr Zpandas.util._exceptionsr Zpandas.core.dtypes.castr Zpandas.core.dtypes.commonrrrZpandas.core.algorithmscorerZpandas.core.arraysrrZpandas.core.commoncommonrZpandas.core.framerZpandas.core.groupbyrZpandas.core.groupby.categoricalrrZpandas.core.indexes.apirrrZpandas.core.seriesrZpandas.io.formats.printingrZpandas.core.genericrrrkrFrrzr-r5r5r5r6sR          | ^