parsel package

Submodules

parsel.csstranslator module

class parsel.csstranslator.GenericTranslator

Bases: parsel.csstranslator.TranslatorMixin, cssselect.xpath.GenericTranslator

css_to_xpath

class parsel.csstranslator.HTMLTranslator(xhtml=False)

Bases: parsel.csstranslator.TranslatorMixin, cssselect.xpath.HTMLTranslator

css_to_xpath

class parsel.csstranslator.TranslatorMixin

Bases: object

This mixin adds support to CSS pseudo elements via dynamic dispatch.

Currently supported pseudo-elements are ::text and ::attr(ATTR_NAME).

xpath_attr_functional_pseudo_element(xpath, function)

Support selecting attribute values using ::attr() pseudo-element

xpath_element(selector)

xpath_pseudo_element(xpath, pseudo_element)

Dispatch method that transforms XPath to support pseudo-element

xpath_text_simple_pseudo_element(xpath)

Support selecting text nodes using ::text pseudo-element

class parsel.csstranslator.XPathExpr(path='', element='*', condition='', star_prefix=False)

Bases: cssselect.xpath.XPathExpr

attribute = None

classmethod from_xpath(xpath, textnode=False, attribute=None)

join(combiner, other)

textnode = False

parsel.csstranslator.css2xpath(query)

Return translated XPath version of a given CSS query

parsel.selector module

XPath selectors based on lxml

class parsel.selector.SafeXMLParser(*args, **kwargs)

Bases: lxml.etree.XMLParser

class parsel.selector.Selector(text=None, type=None, namespaces=None, root=None, base_url=None, _expr=None)

Bases: object

Selector allows you to select parts of an XML or HTML text using CSS or XPath expressions and extract data from it.

text is a unicode object in Python 2 or a str object in Python 3

type defines the selector type, it can be "html", "xml" or None (default). If type is None, the selector defaults to "html".

property attrib

Return the attributes dictionary for underlying element.

css(query)

Apply the given CSS selector and return a SelectorList instance.

query is a string containing the CSS selector to apply.

In the background, CSS queries are translated into XPath queries using cssselect library and run .xpath() method.

extract()

Serialize and return the matched nodes in a single unicode string. Percent encoded content is unquoted.

get()

Serialize and return the matched nodes in a single unicode string. Percent encoded content is unquoted.

getall()

Serialize and return the matched node in a 1-element list of unicode strings.

namespaces

re(regex, replace_entities=True)

Apply the given regex and return a list of unicode strings with the matches.

regex can be either a compiled regular expression or a string which will be compiled to a regular expression using re.compile(regex).

By default, character entity references are replaced by their corresponding character (except for & and <). Passing replace_entities as False switches off these replacements.

re_first(regex, default=None, replace_entities=True)

Apply the given regex and return the first unicode string which matches. If there is no match, return the default value (None if the argument is not provided).

By default, character entity references are replaced by their corresponding character (except for & and <). Passing replace_entities as False switches off these replacements.

register_namespace(prefix, uri)

Register the given namespace to be used in this Selector. Without registering namespaces you can’t select or extract data from non-standard namespaces. See Working on XML (and namespaces).

remove_namespaces()

Remove all namespaces, allowing to traverse the document using namespace-less xpaths. See Removing namespaces.

root

selectorlist_cls

alias of SelectorList

text

type

xpath(query, namespaces=None, **kwargs)

Find nodes matching the xpath query and return the result as a SelectorList instance with all elements flattened. List elements implement Selector interface too.

query is a string containing the XPATH query to apply.

namespaces is an optional prefix: namespace-uri mapping (dict) for additional prefixes to those registered with register_namespace(prefix, uri). Contrary to register_namespace(), these prefixes are not saved for future calls.

Any additional named arguments can be used to pass values for XPath variables in the XPath expression, e.g.:

selector.xpath('//a[href=$url]', url="http://www.example.com")

class parsel.selector.SelectorList

Bases: list

The SelectorList class is a subclass of the builtin list class, which provides a few additional methods.

property attrib

Return the attributes dictionary for the first element. If the list is empty, return an empty dict.

css(query)

Call the .css() method for each element in this list and return their results flattened as another SelectorList.

query is the same argument as the one in Selector.css()

extract()

Call the .get() method for each element is this list and return their results flattened, as a list of unicode strings.

extract_first(default=None)

Return the result of .get() for the first element in this list. If the list is empty, return the default value.

get(default=None)

Return the result of .get() for the first element in this list. If the list is empty, return the default value.

getall()

Call the .get() method for each element is this list and return their results flattened, as a list of unicode strings.

re(regex, replace_entities=True)

Call the .re() method for each element in this list and return their results flattened, as a list of unicode strings.

By default, character entity references are replaced by their corresponding character (except for & and <. Passing replace_entities as False switches off these replacements.

re_first(regex, default=None, replace_entities=True)

Call the .re() method for the first element in this list and return the result in an unicode string. If the list is empty or the regex doesn’t match anything, return the default value (None if the argument is not provided).

By default, character entity references are replaced by their corresponding character (except for & and <. Passing replace_entities as False switches off these replacements.

xpath(xpath, namespaces=None, **kwargs)

Call the .xpath() method for each element in this list and return their results flattened as another SelectorList.

query is the same argument as the one in Selector.xpath()

namespaces is an optional prefix: namespace-uri mapping (dict) for additional prefixes to those registered with register_namespace(prefix, uri). Contrary to register_namespace(), these prefixes are not saved for future calls.

Any additional named arguments can be used to pass values for XPath variables in the XPath expression, e.g.:

selector.xpath('//a[href=$url]', url="http://www.example.com")

parsel.selector.create_root_node(text, parser_cls, base_url=None)

Create root node for text using given parser class.

parsel.utils module

parsel.utils.extract_regex(regex, text, replace_entities=True)

Extract a list of unicode strings from the given text/encoding using the following policies: * if the regex contains a named group called “extract” that will be returned * if the regex contains multiple numbered groups, all those will be returned (flattened) * if the regex doesn’t contain any group the entire regex matching is returned

parsel.utils.flatten(sequence) → list

Returns a single, flat list which contains all elements retrieved from the sequence and all recursively contained sub-sequences (iterables). Examples: >>> [1, 2, [3,4], (5,6)] [1, 2, [3, 4], (5, 6)] >>> flatten([[[1,2,3], (42,None)], [4,5], [6], 7, (8,9,10)]) [1, 2, 3, 42, None, 4, 5, 6, 7, 8, 9, 10] >>> flatten([“foo”, “bar”]) [‘foo’, ‘bar’] >>> flatten([“foo”, [“baz”, 42], “bar”]) [‘foo’, ‘baz’, 42, ‘bar’]

parsel.utils.iflatten(sequence) → iterator

Similar to .flatten(), but returns iterator instead

parsel.utils.shorten(text, width, suffix='...')

Truncate the given text to fit in the given width.

Module contents

Parsel lets you extract text from XML/HTML documents using XPath or CSS selectors