SQLAlchemy 0.7.1 Documentation

Version: 0.7.1 Last Updated: 06/05/2011 13:13:59

Relationship Loading Techniques

A big part of SQLAlchemy is providing a wide range of control over how related objects get loaded when querying. This behavior can be configured at mapper construction time using the lazy parameter to the relationship() function, as well as by using options with the Query object.

Using Loader Strategies: Lazy Loading, Eager Loading

By default, all inter-object relationships are lazy loading. The scalar or collection attribute associated with a relationship() contains a trigger which fires the first time the attribute is accessed. This trigger, in all but one case, issues a SQL call at the point of access in order to load the related object or objects:

sql>>> jack.addresses
[<Address(u'jack@google.com')>, <Address(u'j25@yahoo.com')>]

The one case where SQL is not emitted is for a simple many-to-one relationship, when the related object can be identified by its primary key alone and that object is already present in the current Session.

This default behavior of “load upon attribute access” is known as “lazy” or “select” loading - the name “select” because a “SELECT” statement is typically emitted when the attribute is first accessed.

In the Object Relational Tutorial, we introduced the concept of Eager Loading. We used an option in conjunction with the Query object in order to indicate that a relationship should be loaded at the same time as the parent, within a single SQL query. This option, known as joinedload(), connects a JOIN (by default a LEFT OUTER join) to the statement and populates the scalar/collection from the same result set as that of the parent:

sql>>> jack = session.query(User).options(joinedload('addresses')).filter_by(name='jack').all() 

In addition to “joined eager loading”, a second option for eager loading exists, called “subquery eager loading”. This kind of eager loading emits an additional SQL statement for each collection requested, aggregated across all parent objects:

sql>>>jack = session.query(User).options(subqueryload('addresses')).filter_by(name='jack').all()

The default loader strategy for any relationship() is configured by the lazy keyword argument, which defaults to select - this indicates a “select” statement . Below we set it as joined so that the children relationship is eager loading, using a join:

# load the 'children' collection using LEFT OUTER JOIN
mapper(Parent, parent_table, properties={
    'children': relationship(Child, lazy='joined')
})

We can also set it to eagerly load using a second query for all collections, using subquery:

# load the 'children' attribute using a join to a subquery
mapper(Parent, parent_table, properties={
    'children': relationship(Child, lazy='subquery')
})

When querying, all three choices of loader strategy are available on a per-query basis, using the joinedload(), subqueryload() and lazyload() query options:

# set children to load lazily
session.query(Parent).options(lazyload('children')).all()

# set children to load eagerly with a join
session.query(Parent).options(joinedload('children')).all()

# set children to load eagerly with a second statement
session.query(Parent).options(subqueryload('children')).all()

To reference a relationship that is deeper than one level, separate the names by periods:

session.query(Parent).options(joinedload('foo.bar.bat')).all()

When using dot-separated names with joinedload() or subqueryload(), the option applies only to the actual attribute named, and not its ancestors. For example, suppose a mapping from A to B to C, where the relationships, named atob and btoc, are both lazy-loading. A statement like the following:

session.query(A).options(joinedload('atob.btoc')).all()

will load only A objects to start. When the atob attribute on each A is accessed, the returned B objects will eagerly load their C objects.

Therefore, to modify the eager load to load both atob as well as btoc, place joinedloads for both:

session.query(A).options(joinedload('atob'), joinedload('atob.btoc')).all()

or more simply just use joinedload_all() or subqueryload_all():

session.query(A).options(joinedload_all('atob.btoc')).all()

There are two other loader strategies available, dynamic loading and no loading; these are described in Working with Large Collections.

The Zen of Eager Loading

The philosophy behind loader strategies is that any set of loading schemes can be applied to a particular query, and the results don’t change - only the number of SQL statements required to fully load related objects and collections changes. A particular query might start out using all lazy loads. After using it in context, it might be revealed that particular attributes or collections are always accessed, and that it would be more efficient to change the loader strategy for these. The strategy can be changed with no other modifications to the query, the results will remain identical, but fewer SQL statements would be emitted. In theory (and pretty much in practice), nothing you can do to the Query would make it load a different set of primary or related objects based on a change in loader strategy.

The way eagerloading does this, and in particular how joinedload() works, is that it creates an anonymous alias of all the joins it adds to your query, so that they can’t be referenced by other parts of the query. If the query contains a DISTINCT, or a limit or offset, the statement is first wrapped inside a subquery, and joins are applied to that. As the user, you don’t have access to these aliases or subqueries, and you cannot affect what data they will load at query time - a typical beginner misunderstanding is that adding a Query.order_by(), naming the joined relationship, would change the order of the collection, or that the entries in the collection as it is loaded could be affected by Query.filter(). Not the case ! If you’d like to join from one table to another, filtering or ordering on the joined result, you’d use Query.join(). If you then wanted that joined result to populate itself into a related collection, this is also available, via contains_eager() option - see Routing Explicit Joins/Statements into Eagerly Loaded Collections.

What Kind of Loading to Use ?

Which type of loading to use typically comes down to optimizing the tradeoff between number of SQL executions, complexity of SQL emitted, and amount of data fetched. Lets take two examples, a relationship() which references a collection, and a relationship() that references a scalar many-to-one reference.

  • One to Many Collection
  • When using the default lazy loading, if you load 100 objects, and then access a collection on each of them, a total of 101 SQL statements will be emitted, although each statement will typically be a simple SELECT without any joins.
  • When using joined loading, the load of 100 objects and their collections will emit only one SQL statement. However, the total number of rows fetched will be equal to the sum of the size of all the collections, plus one extra row for each parent object that has an empty collection. Each row will also contain the full set of columns represented by the parents, repeated for each collection item - SQLAlchemy does not re-fetch these columns other than those of the primary key, however most DBAPIs (with some exceptions) will transmit the full data of each parent over the wire to the client connection in any case. Therefore joined eager loading only makes sense when the size of the collections are relatively small. The LEFT OUTER JOIN can also be performance intensive compared to an INNER join.
  • When using subquery loading, the load of 100 objects will emit two SQL statements. The second statement will fetch a total number of rows equal to the sum of the size of all collections. An INNER JOIN is used, and a minimum of parent columns are requested, only the primary keys. So a subquery load makes sense when the collections are larger.
  • When multiple levels of depth are used with joined or subquery loading, loading collections-within- collections will multiply the total number of rows fetched in a cartesian fashion. Both forms of eager loading always join from the original parent class.
  • Many to One Reference
  • When using the default lazy loading, a load of 100 objects will like in the case of the collection emit as many as 101 SQL statements. However - there is a significant exception to this, in that if the many-to-one reference is a simple foreign key reference to the target’s primary key, each reference will be checked first in the current identity map using query.get(). So here, if the collection of objects references a relatively small set of target objects, or the full set of possible target objects have already been loaded into the session and are strongly referenced, using the default of lazy=’select’ is by far the most efficient way to go.
  • When using joined loading, the load of 100 objects will emit only one SQL statement. The join will be a LEFT OUTER JOIN, and the total number of rows will be equal to 100 in all cases. If you know that each parent definitely has a child (i.e. the foreign key reference is NOT NULL), the joined load can be configured with innerjoin=True, which is usually specified within the relationship(). For a load of objects where there are many possible target references which may have not been loaded already, joined loading with an INNER JOIN is extremely efficient.
  • Subquery loading will issue a second load for all the child objects, so for a load of 100 objects there would be two SQL statements emitted. There’s probably not much advantage here over joined loading, however, except perhaps that subquery loading can use an INNER JOIN in all cases whereas joined loading requires that the foreign key is NOT NULL.

Routing Explicit Joins/Statements into Eagerly Loaded Collections

The behavior of joinedload() is such that joins are created automatically, the results of which are routed into collections and scalar references on loaded objects. It is often the case that a query already includes the necessary joins which represent a particular collection or scalar reference, and the joins added by the joinedload feature are redundant - yet you’d still like the collections/references to be populated.

For this SQLAlchemy supplies the contains_eager() option. This option is used in the same manner as the joinedload() option except it is assumed that the Query will specify the appropriate joins explicitly. Below it’s used with a from_statement load:

# mapping is the users->addresses mapping
mapper(User, users_table, properties={
    'addresses': relationship(Address, addresses_table)
})

# define a query on USERS with an outer join to ADDRESSES
statement = users_table.outerjoin(addresses_table).select().apply_labels()

# construct a Query object which expects the "addresses" results
query = session.query(User).options(contains_eager('addresses'))

# get results normally
r = query.from_statement(statement)

It works just as well with an inline Query.join() or Query.outerjoin():

session.query(User).outerjoin(User.addresses).options(contains_eager(User.addresses)).all()

If the “eager” portion of the statement is “aliased”, the alias keyword argument to contains_eager() may be used to indicate it. This is a string alias name or reference to an actual Alias (or other selectable) object:

# use an alias of the Address entity
adalias = aliased(Address)

# construct a Query object which expects the "addresses" results
query = session.query(User).\
    outerjoin(adalias, User.addresses).\
    options(contains_eager(User.addresses, alias=adalias))

# get results normally
sqlr = query.all()

The alias argument is used only as a source of columns to match up to the result set. You can use it even to match up the result to arbitrary label names in a string SQL statement, by passing a selectable() which links those labels to the mapped Table:

# label the columns of the addresses table
eager_columns = select([
                    addresses.c.address_id.label('a1'),
                    addresses.c.email_address.label('a2'),
                    addresses.c.user_id.label('a3')])

# select from a raw SQL statement which uses those label names for the
# addresses table.  contains_eager() matches them up.
query = session.query(User).\
    from_statement("select users.*, addresses.address_id as a1, "
            "addresses.email_address as a2, addresses.user_id as a3 "
            "from users left outer join addresses on users.user_id=addresses.user_id").\
    options(contains_eager(User.addresses, alias=eager_columns))

The path given as the argument to contains_eager() needs to be a full path from the starting entity. For example if we were loading Users->orders->Order->items->Item, the string version would look like:

query(User).options(contains_eager('orders', 'items'))

Or using the class-bound descriptor:

query(User).options(contains_eager(User.orders, Order.items))

A variant on contains_eager() is the contains_alias() option, which is used in the rare case that the parent object is loaded from an alias within a user-defined SELECT statement:

# define an aliased UNION called 'ulist'
statement = users.select(users.c.user_id==7).union(users.select(users.c.user_id>7)).alias('ulist')

# add on an eager load of "addresses"
statement = statement.outerjoin(addresses).select().apply_labels()

# create query, indicating "ulist" is an alias for the main table, "addresses" property should
# be eager loaded
query = session.query(User).options(contains_alias('ulist'), contains_eager('addresses'))

# results
r = query.from_statement(statement)

Relation Loader API

sqlalchemy.orm.contains_alias(alias)

Return a MapperOption that will indicate to the query that the main table has been aliased.

alias is the string name or Alias object representing the alias.

sqlalchemy.orm.contains_eager(*keys, **kwargs)

Return a MapperOption that will indicate to the query that the given attribute should be eagerly loaded from columns currently in the query.

Used with options().

The option is used in conjunction with an explicit join that loads the desired rows, i.e.:

sess.query(Order).\
        join(Order.user).\
        options(contains_eager(Order.user))

The above query would join from the Order entity to its related User entity, and the returned Order objects would have the Order.user attribute pre-populated.

contains_eager() also accepts an alias argument, which is the string name of an alias, an alias() construct, or an aliased() construct. Use this when the eagerly-loaded rows are to come from an aliased table:

user_alias = aliased(User)
sess.query(Order).\
        join((user_alias, Order.user)).\
        options(contains_eager(Order.user, alias=user_alias))

See also eagerload() for the “automatic” version of this functionality.

For additional examples of contains_eager() see Routing Explicit Joins/Statements into Eagerly Loaded Collections.

sqlalchemy.orm.eagerload(*args, **kwargs)

A synonym for joinedload().

sqlalchemy.orm.eagerload_all(*args, **kwargs)

A synonym for joinedload_all()

sqlalchemy.orm.joinedload(*keys, **kw)

Return a MapperOption that will convert the property of the given name or series of mapped attributes into an joined eager load.

Note

This function is known as eagerload() in all versions of SQLAlchemy prior to version 0.6beta3, including the 0.5 and 0.4 series. eagerload() will remain available for the foreseeable future in order to enable cross-compatibility.

Used with options().

examples:

# joined-load the "orders" colleciton on "User"
query(User).options(joinedload(User.orders))

# joined-load the "keywords" collection on each "Item",
# but not the "items" collection on "Order" - those 
# remain lazily loaded.
query(Order).options(joinedload(Order.items, Item.keywords))

# to joined-load across both, use joinedload_all()
query(Order).options(joinedload_all(Order.items, Item.keywords))

joinedload() also accepts a keyword argument innerjoin=True which indicates using an inner join instead of an outer:

query(Order).options(joinedload(Order.user, innerjoin=True))

Note that the join created by joinedload() is aliased such that no other aspects of the query will affect what it loads. To use joined eager loading with a join that is constructed manually using join() or join(), see contains_eager().

See also: subqueryload(), lazyload()

sqlalchemy.orm.joinedload_all(*keys, **kw)

Return a MapperOption that will convert all properties along the given dot-separated path or series of mapped attributes into an joined eager load.

Note

This function is known as eagerload_all() in all versions of SQLAlchemy prior to version 0.6beta3, including the 0.5 and 0.4 series. eagerload_all() will remain available for the foreseeable future in order to enable cross-compatibility.

Used with options().

For example:

query.options(joinedload_all('orders.items.keywords'))...

will set all of ‘orders’, ‘orders.items’, and ‘orders.items.keywords’ to load in one joined eager load.

Individual descriptors are accepted as arguments as well:

query.options(joinedload_all(User.orders, Order.items, Item.keywords))

The keyword arguments accept a flag innerjoin=True|False which will override the value of the innerjoin flag specified on the relationship().

See also: subqueryload_all(), lazyload()

sqlalchemy.orm.lazyload(*keys)

Return a MapperOption that will convert the property of the given name or series of mapped attributes into a lazy load.

Used with options().

See also: eagerload(), subqueryload(), immediateload()

sqlalchemy.orm.subqueryload(*keys)

Return a MapperOption that will convert the property of the given name or series of mapped attributes into an subquery eager load.

Used with options().

examples:

# subquery-load the "orders" colleciton on "User"
query(User).options(subqueryload(User.orders))

# subquery-load the "keywords" collection on each "Item",
# but not the "items" collection on "Order" - those 
# remain lazily loaded.
query(Order).options(subqueryload(Order.items, Item.keywords))

# to subquery-load across both, use subqueryload_all()
query(Order).options(subqueryload_all(Order.items, Item.keywords))

See also: joinedload(), lazyload()

sqlalchemy.orm.subqueryload_all(*keys)

Return a MapperOption that will convert all properties along the given dot-separated path or series of mapped attributes into a subquery eager load.

Used with options().

For example:

query.options(subqueryload_all('orders.items.keywords'))...

will set all of ‘orders’, ‘orders.items’, and ‘orders.items.keywords’ to load in one subquery eager load.

Individual descriptors are accepted as arguments as well:

query.options(subqueryload_all(User.orders, Order.items,
Item.keywords))

See also: joinedload_all(), lazyload(), immediateload()

Previous: Querying Next: ORM Events