nuclear_eclipse 2 months ago

Author here: I created Bowler as a "hackamonth" project when I joined Facebook's internal Python Foundation team. We've already used it for a bunch of random codemods that touch a large number of source files throughout our codebase.

Happy to answer any questions you might have!

  • Cynddl 2 months ago

    How does it compare to redbaron? [1]

    > RedBaron is a python library and tool powerful enough to be used into IPython solely that intent to make the process of writing code that modify source code as easy and as simple as possible. That include writing custom refactoring, generic refactoring, tools, IDE or directly modifying you source code into IPython with a higher and more powerful abstraction than the advanced texts modification tools that you find in advanced text editors and IDE.

    [1] https://github.com/PyCQA/redbaron

    • nuclear_eclipse 2 months ago

      I haven't actually used RedBaron, but from glancing at the docs, it would look like there are two primary differences:

      1) RedBaron uses a procedural API that modifies state continuously as you call methods on the class, whereas Bowler allows you expressive a series of queries/transforms up front, and then execute those all at once on your entire codebase. This also allows Bowler to provide a fluent API, so that you can chain all method calls from each other.

      2) RedBaron uses a custom AST implementation, which still lists "Python 3.7 support" on its roadmap. Building on "fissix" (a backport of lib2to3) means that Bowler had day one support for Python 3.7 syntax, and already supports the provisional Python 3.8 syntax. This means your refactoring tools will never prevent you from adopting new versions of Python as soon as they are released.

typon 2 months ago

Could this be used to create a macro system for python?

MrSaints 2 months ago

This is akin to the other "codemod" facilities Facebook already uses for large-scale refactoring in busy codebases (particularly, their JS codebases), but for Python (https://github.com/facebook/codemod, https://github.com/facebook/jscodeshift).

  • nuclear_eclipse 2 months ago

    Notably, or notoriously, our previous codemod project just used regexes, which could result in ballooning complexity, especially when needing to modify code that might include type annotations. Bowler was designed specifically to allow refactoring against more complicated subjects, such as function signature changes, where you lose all predictability in formatting at both the definition and call site.

jsmeaton 2 months ago

I could imagine a library of transformers being created using this tool.

For instance we’ve just enabled the pep3101 Flake8 plugin which enforces newstyle string formatting over % formatting. I’d love to see a transformer that automates that refactoring.

  • nuclear_eclipse 2 months ago

    This sounds like exactly the sort of transforms and codemods that Bowler was designed to handle. Furthermore, we plan to support more linter-style features, and would like to have it integrated with tools like Flake8 or Phabricator, so that it can simultaneously find lints, and immediately suggest modifications to resolve those lints.

ausjke 2 months ago

Is this to help migration from python 2 to python 3, or a tool to make python 3 better? it's unclear after I read it except it mentions it's based on 2to3.

  • jwilk 2 months ago

    Bowler doesn't have much to do with 2to3, except that they have the same library underneath.

    lib2to3, despite the name, is a fairly generic refactoring library.

  • nuclear_eclipse 2 months ago

    It was not built for the 2 to 3 migration, as there are plenty of great tools already out there for that, like 2to3 itself or futurize. It was built more for cases where we want to make changes to things like function or method signatures, and update both the definition and all call sites, across our entire codebase. It's still a very rudimentary tool at this point, but it was designed in a way that allows us to check in and reuse many components of the codemod, so that we can extract longer term value from these tools.

marmaduke 2 months ago

How well does this handle dynamic constructs? Eg if I rename a method and elsewhere have

    if hasattr(obj, ‘foo’):
does that get caught or not?
  • nuclear_eclipse 2 months ago

    No, it would not catch that, because it's not tracing elements through the AST. It would be possible to do some amount of tracing, but it would be impossible to catch and modify all variants of that, since Python is an extremely dynamic language. Imagine something like:

        attr = "foo" if conditionA else "bar"
        obj = A() if conditionB else B()
        if hasattr(obj, attr):
            ...
    
    Because of this, Bowler is focused on being more practical, and leans on the expectation that an engineer will be in the loop and validate the resulting diff, run unit tests, etc.
    • figgis 2 months ago

      I apologize but I'm really having trouble understanding what the problem you are referring too. In python syntax that would be written as -

          if ConditionA:
              attr = "foo"
          else:
              attr = "bar"
      
          if ConditionB:
              obj = A()
          else:
              obj = B()
      
          if hasattr(attr, obj):
              do_magic()
      
      I haven't tested Bowler yet but I would think something that refactors code would handle that pretty easily?
      • ndr 2 months ago

        A() and B() could return two different classes that both support foo and bar methods. If your refactor is to rename one of those methods for one of the classes this code breaks and it's not trivial to update automatically, is it?

        • shoo 2 months ago

          in general, given this is python, A and B could do something completely foul like request data from the internets and `eval(...)` it then return the result, or redefine `True, False = False, True` or redefine

            hasattr = lambda *args: False`
          
          or something equally insane and difficult to statically analyse

          > Eg if I rename a method and elsewhere have `if hasattr(obj, ‘foo’):`

          Further, even in the absence of insane dynamic code, it probably isn't possible to automatically make a correct decision without knowing the intent of the original code.

          For example, maybe we're renaming the `foo` method of `Fizz` class to `barr`, and currently the program is written such that `is hasattr(obj, 'foo'):` only happens to be called on `Fizz` instances. Is it the case that the programmer only ever intended that code to be called on `Fizz` instances, or did the programmer intend it should operate on all types that have a `foo` attribute, including old-style `Fizz` instances and new types that might be encountered in future? In the former case, we should rewrite the logic to `hasattr(obj, 'barr')`, in the latter case perhaps we should leave it unchanged, so it will no longer trigger after the refactor.

          i do agree it should be possible to automatically flag this as "here be something interesting to consider! please intervene and make a manual decision!"

          • marmaduke 2 months ago

            I think these sorts of cases of obfuscated code are outside the scope of refactoring tools. No one who’s looking to use a refactoring tool has to write or read code that uses eval or redefines True.

      • zbentley 2 months ago

        > I would think something that refactors code would handle that pretty easily?

        Nope. Something that refactors code could potentially identify that as an ambiguous case reliably, but it's impossible to statically refactor code of that sort in the general case. Without external/runtime behavior verification/specification, that is.

      • falcor84 2 months ago

        >In python syntax that would be ...

        That was Python syntax.

        • figgis 2 months ago

          See below

          • _alexh_AS_ 2 months ago

              Python 3.6.6 (tags/v3.6.6:4cf1f54eb7, Jul  6 2018, 15:35:19) 
              [GCC 4.8.4] on linux
              Type "help", "copyright", "credits" or "license" for more information.
              >>> conditionA = True
              >>> attr = "foo" if conditionA else "bar"
              >>> print(attr)
              foo
              >>> conditionA = False
              >>> attr = "foo" if conditionA else "bar"
              >>> print(attr)
              bar
              >>>
sandGorgon 2 months ago

Is it possible to use this in vscode ?

That would be awesome.

  • nuclear_eclipse 2 months ago

    The possibility is there, but we haven't pursued any specific integrations yet. For obvious reasons, we would most likely work on Nuclide integration first, and something like VSCode support would probably need to come from the community. As Bowler is primarily providing the API framework, most integrations will likely just be plugins that make calls to the Bowler API, or execute the bowler CLI, with appropriate queries.

  • jamespo 2 months ago

    ditto but for emacs

linuxftw 2 months ago

Seems like more work to me. Simple search and replace is usually sufficient for something like what the intro video shows.

  • gh02t 2 months ago

    This looks tedious to use directly, but it'd be good to build things like editor plugins on top of. The docs also show that it can do stuff that would not be simple to do with search and replace (e.g. transforming function arguments).

    • stcredzero 2 months ago

      The original Smalltalk refactoring engine didn't take off until it was packaged as the Refactoring Browser. But the real utility in it was making arbitrary scripting of meta-level and syntax driven code transformation very accessible. You can literally be an order of magnitude more powerful with refactorings with a tool like that.

      Python is a good language for for this sort of tool. There is a lot of potential for the power of Python harnessed to rewrite Python.

  • nuclear_eclipse 2 months ago

    For something that simple, I would agree, but the demo is barely scratching the surface. I'd like to make a more in-depth version showing a more complicated transform, such as changing a function signature and having it update the callers across multiple files.

    • linuxftw 2 months ago

      Yeah, maybe some other example would be more convincing. I think this case would still be a good job for search and replace, you'll need to investigate the surrounding code to get the new data to pass to the function or modify the existing data to support the new format. It's unlikely that you add or remove an argument from a function and nothing else needs to change.

  • SketchySeaBeast 2 months ago

    I'm finding Visual Studio Code does an excellent job with "rename symbol" or "change all occurrences".

    • drb91 2 months ago

      Funny, I've found it seems to struggle with any references that cross language boundaries.... except with typescript, for some reason.

      EDIT: Sorry, I meant 'file' boundary, not 'language' boundary.

      • pvg 2 months ago

        Why not, you know, edit the actual sentence instead of adding a thing that says EDIT but leaves the original confusingly unedited?

        • yesenadam 2 months ago

          Because, I guess, it's courteous to the people who already commented on the original, so they don't look like they were talking nonsense.

          • pvg 2 months ago

            You make the edit and then you describe the edit, to avoid that.

            • yesenadam 2 months ago

              Ah, good point. (I guess I wasn't confused by the edit - though if I'd been here longer I probably would've been - but I was by your comment!)

              • pvg 2 months ago

                Wait, what? You can't just politely walk away from a from a pointless argument over minutia like that. You are new around here. These must extend to a minimum thread depth of 12. Please review the guidelines.

      • SketchySeaBeast 2 months ago

        Mine was in reference to Python projects - I haven't tried it on cross language references.