Python: Modernize 4 queries for missing/multiple calls to init/del methods

[joefarebrother]

Modernizes py/missing-call-to-init, py/missing-call-to-del, py/multiple-calls-to-init,, and py/multiplecalls-to-del.
Uses DataFlowDispatch methods rather than pointsTo.

Improves precision (to reduce FPs) in resolving relevant calls as precisely as possible using known MRO information, improves descriptiveness of alert messages by including more possible relevant locations, and improves documentation.

[github-actions]

QHelp previews:

python/ql/src/Classes/CallsToInitDel/MissingCallToDel.qhelp

Missing call to superclass __del__ during object destruction

Python, unlike some other object-oriented languages such as Java, allows the developer complete freedom in when and how superclass finalizers are called during object finalization. However, the developer has responsibility for ensuring that objects are properly cleaned up, and that all superclass __del__ methods are called.

Classes with a __del__ method (a finalizer) typically hold some resource such as a file handle that needs to be cleaned up. If the __del__ method of a superclass is not called during object finalization, it is likely that resources may be leaked.

A call to the __del__ method of a superclass during object initialization may be unintentionally skipped:

• If a subclass calls the __del__ method of the wrong class.
• If a call to the __del__ method of one its base classes is omitted.
• If a call to super().__del__ is used, but not all __del__ methods in the Method Resolution Order (MRO) chain themselves call super(). This in particular arises more often in cases of multiple inheritance.

Recommendation

Ensure that all superclass __del__ methods are properly called. Either each base class's finalize method should be explicitly called, or super() calls should be consistently used throughout the inheritance hierarchy.

Example

In the following example, explicit calls to __del__ are used, but SportsCar erroneously calls Vehicle.__del__. This is fixed in FixedSportsCar by calling Car.__del__.

class Vehicle(object):
    
    def __del__(self):
        recycle(self.base_parts)
        
class Car(Vehicle):
    
    def __del__(self):
        recycle(self.car_parts)
        Vehicle.__del__(self)
        
#BAD: Car.__del__ is not called.
class SportsCar(Car, Vehicle):
    
    def __del__(self):
        recycle(self.sports_car_parts)
        Vehicle.__del__(self)
        
#GOOD: Car.__del__ is called correctly.
class FixedSportsCar(Car, Vehicle):
    
    def __del__(self):
        recycle(self.sports_car_parts)
        Car.__del__(self)
        

References

• Python Reference: del.
• Python Standard Library: super.
• Python Glossary: Method resolution order.

python/ql/src/Classes/CallsToInitDel/MissingCallToInit.qhelp

Missing call to superclass __init__ during object initialization

Python, unlike some other object-oriented languages such as Java, allows the developer complete freedom in when and how superclass initializers are called during object initialization. However, the developer has responsibility for ensuring that objects are properly initialized, and that all superclass __init__ methods are called.

If the __init__ method of a superclass is not called during object initialization, this can lead to errors due to the object not being fully initialized, such as having missing attributes.

A call to the __init__ method of a superclass during object initialization may be unintentionally skipped:

• If a subclass calls the __init__ method of the wrong class.
• If a call to the __init__ method of one its base classes is omitted.
• If a call to super().__init__ is used, but not all __init__ methods in the Method Resolution Order (MRO) chain themselves call super(). This in particular arises more often in cases of multiple inheritance.

Recommendation

Ensure that all superclass __init__ methods are properly called. Either each base class's initialize method should be explicitly called, or super() calls should be consistently used throughout the inheritance hierarchy.

Example

In the following example, explicit calls to __init__ are used, but SportsCar erroneously calls Vehicle.__init__. This is fixed in FixedSportsCar by calling Car.__init__.

class Vehicle(object):
    
    def __init__(self):
        self.mobile = True
        
class Car(Vehicle):
    
    def __init__(self):
        Vehicle.__init__(self)
        self.car_init()
        
# BAD: Car.__init__ is not called.
class SportsCar(Car, Vehicle):
    
    def __init__(self):
        Vehicle.__init__(self)
        self.sports_car_init()
        
# GOOD: Car.__init__ is called correctly.
class FixedSportsCar(Car, Vehicle):
    
    def __init__(self):
        Car.__init__(self)
        self.sports_car_init()
        

References

• Python Reference: init.
• Python Standard Library: super.
• Python Glossary: Method resolution order.

python/ql/src/Classes/CallsToInitDel/SuperclassDelCalledMultipleTimes.qhelp

Multiple calls to __del__ during object destruction

Python, unlike some other object-oriented languages such as Java, allows the developer complete freedom in when and how superclass finalizers are called during object finalization. However, the developer has responsibility for ensuring that objects are properly cleaned up.

Objects with a __del__ method (a finalizer) often hold resources such as file handles that need to be cleaned up. If a superclass finalizer is called multiple times, this may lead to errors such as closing an already closed file, and lead to objects not being cleaned up properly as expected.

There are a number of ways that a __del__ method may be be called more than once.

• There may be more than one explicit call to the method in the hierarchy of __del__ methods.
• In situations involving multiple inheritance, an finalization method may call the finalizers of each of its base types, which themselves both call the finalizer of a shared base type. (This is an example of the Diamond Inheritance problem)
• Another situation involving multiple inheritance arises when a subclass calls the __del__ methods of each of its base classes, one of which calls super().__del__. This super call resolves to the next class in the Method Resolution Order (MRO) of the subclass, which may be another base class that already has its initializer explicitly called.

Recommendation

Ensure that each finalizer method is called exactly once during finalization. This can be ensured by calling super().__del__ for each finalizer method in the inheritance chain.

Example

In the following example, there is a mixture of explicit calls to __del__ and calls using super(), resulting in Vehicle.__del__ being called twice. FixedSportsCar.__del__ fixes this by using super() consistently with the other delete methods.

#Calling a method multiple times by using explicit calls when a base uses super()
class Vehicle(object):
     
    def __del__(self):
        recycle(self.base_parts)
        super(Vehicle, self).__del__()
        
class Car(Vehicle):
    
    def __del__(self):
        recycle(self.car_parts)
        super(Car, self).__del__()
        
        
class SportsCar(Car, Vehicle):
    
    # BAD: Vehicle.__del__ will get called twice
    def __del__(self):
        recycle(self.sports_car_parts)
        Car.__del__(self)
        Vehicle.__del__(self)
        
        
# GOOD: super() is used ensuring each del method is called once.
class FixedSportsCar(Car, Vehicle):
    
    def __del__(self):
        recycle(self.sports_car_parts)
        super(SportsCar, self).__del__()
     

References

• Python Reference: del.
• Python Standard Library: super.
• Python Glossary: Method resolution order.
• Wikipedia: The Diamond Problem.

python/ql/src/Classes/CallsToInitDel/SuperclassInitCalledMultipleTimes.qhelp

Multiple calls to __init__ during object initialization

Python, unlike some other object-oriented languages such as Java, allows the developer complete freedom in when and how superclass initializers are called during object initialization. However, the developer has responsibility for ensuring that objects are properly initialized.

Calling an __init__ method more than once during object initialization risks the object being incorrectly initialized, as the method and the rest of the inheritance chain may not have been written with the expectation that it could be called multiple times. For example, it may set attributes to a default value in a way that unexpectedly overwrites values setting those attributes in a subclass.

There are a number of ways that an __init__ method may be be called more than once.

• There may be more than one explicit call to the method in the hierarchy of __init__ methods.
• In situations involving multiple inheritance, an initialization method may call the initializers of each of its base types, which themselves both call the initializer of a shared base type. (This is an example of the Diamond Inheritance problem)
• Another situation involving multiple inheritance arises when a subclass calls the __init__ methods of each of its base classes, one of which calls super().__init__. This super call resolves to the next class in the Method Resolution Order (MRO) of the subclass, which may be another base class that already has its initializer explicitly called.

Recommendation

Take care whenever possible not to call an an initializer multiple times. If each __init__ method in the hierarchy calls super().__init__(), then each initializer will be called exactly once according to the MRO of the subclass. When explicitly calling base class initializers (such as to pass different arguments to different initializers), ensure this is done consistently throughout, rather than using super() calls in the base classes.

In some cases, it may not be possible to avoid calling a base initializer multiple times without significant refactoring. In this case, carefully check that the initializer does not interfere with subclass initializers when called multiple times (such as by overwriting attributes), and ensure this behavior is documented.

Example

In the following (BAD) example, the class D calls B.__init__ and C.__init__, which each call A.__init__. This results in self.state being set to None as A.__init__ is called again after B.__init__ had finished. This may lead to unexpected results.

class A:
    def __init__(self):
        self.state = None 

class B(A):
    def __init__(self):
        A.__init__(self)
        self.state = "B"
        self.b = 3 

class C(A):
    def __init__(self):
        A.__init__(self)
        self.c = 2 

class D(B,C):
    def __init__(self):
        B.__init__(self)
        C.__init__(self) # BAD: This calls A.__init__ a second time, setting self.state to None.
        

In the following (GOOD) example, a call to super().__init__ is made in each class in the inheritance hierarchy, ensuring each initializer is called exactly once.

class A:
    def __init__(self):
        self.state = None 

class B(A):
    def __init__(self):
        super().__init__()
        self.state = "B"
        self.b = 3 

class C(A):
    def __init__(self):
        super().__init__()
        self.c = 2 

class D(B,C):
    def __init__(self): # GOOD: Each method calls super, so each init method runs once. self.state will be set to "B".
        super().__init__()
        self.d = 1
        

In the following (BAD) example, explicit base class calls are mixed with super() calls, and C.__init__ is called twice.

class A:
    def __init__(self):
        print("A")
        self.state = None 

class B(A):
    def __init__(self):
        print("B")
        super().__init__() # When called from D, this calls C.__init__
        self.state = "B"
        self.b = 3 

class C(A):
    def __init__(self):
        print("C")
        super().__init__()
        self.c = 2 

class D(B,C):
    def __init__(self): 
        B.__init__(self)
        C.__init__(self) # BAD: C.__init__ is called a second time

References

• Python Reference: init.
• Python Standard Library: super.
• Python Glossary: Method resolution order.
• Wikipedia: The Diamond Problem.

[tausbn]

I think overall this looks really solid, but I must admit I didn't have time to fully digest the new implementation. (However, as I'm going away on PTO, I figured it was best to give you my interim review regardless.)

I'm slightly worried about the many not exists(...) constructions in the modelling (as these may result in large joins). Make sure the performance testing is run with the tuple-counting option enabled.

Other than that, if the performance looks good, I would be happy to merge this.

[tausbn]

This MRO business is a bit awkward. I wonder if we could clean it up by creating a new IPA type representing "the MRO starting at a particular base class", and thus avoid having to thread mroBase through everything (which might mean we could use the built-in transitive closure operators).

This is just some musing on my part -- not necessarily an immediately actionable suggestion.

[joefarebrother]

That could be a useful component of a public-facing dataflow dispatch / call graph resolution API

[tausbn]

One minor comment, otherwise this looks good to me. 👍
I assume DCA was uneventful?

[tausbn]

Hmm... I guess if you have $n$ calls to a superclass method, then this will produce $n\cdot(n-1)/2$ results, though I guess this might be rare in general.

As an alternative, we could consider ranking the superclass method calls by location, and picking the first two items in this ranking. That way, we only ever produce two method calls.

[joefarebrother]

As this query makes use of an Option type with a location, a pattern I've also identified as useful in another quality query, it has been extracted into a Shared Library PR (#20280), that I intend to refactor a little to include once it's merged.

It seems like the previous DCA run had a FreeCAD failure that I'm not certain the cause of. That has historically been indicative of a performance concern in uncommon scenarios. I will start a new run on an updated branch to check whether it reproduces.

[Copilot]

Pull Request Overview

This PR modernizes 4 queries related to missing/multiple calls to __init__/__del__ methods in Python class hierarchies. The queries are moved to a new directory structure and rewritten to use DataFlowDispatch methods instead of the legacy pointsTo approach, improving precision and descriptiveness of alerts.

Key changes:

• Migrates from legacy libraries to modern dataflow analysis for better precision
• Reorganizes query locations from Classes/ to Classes/CallsToInitDel/ directory
• Enhances alert messages with more descriptive information and additional context about method calls

Reviewed Changes

Copilot reviewed 48 out of 48 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
python/ql/test/query-tests/Classes/multiple/multiple-init/multiple_init.py	Updated test cases with more concrete examples and better alerts
python/ql/test/query-tests/Classes/missing-init/missing_init.py	Enhanced test coverage with realistic scenarios and improved alert annotations
python/ql/src/Classes/CallsToInitDel/SuperclassInitCalledMultipleTimes.ql	New modernized query for detecting multiple init calls using dataflow analysis
python/ql/src/Classes/CallsToInitDel/MethodCallOrder.qll	New library providing core logic for method call analysis with MRO support
python/ql/src/Classes/MethodCallOrder.qll	Deprecated the old library in favor of the new implementation
Various .expected files	Updated to reflect new query output format and improved precision